[Paper Reding] Discovering Discrete Latent Topics with Neural Variational Inference

Problem Definition

Develop a fast, accurate and expressive topic model for documents

Introduction

Latent semantic analysis (LSA)

Probabilistic topic models (e.g. PLSA , LDA and HDPs)

• Robust, scalable, and theoretically sound foundation for document modeling
• Introduce latent variables for each token to topic assignment

Traditional Dirichlet-Multinomial topic model

• Inference becomes more complex as topic models have grown more expressive

Deep neural networks

• Freat potential for learning complicated non-linear distributions for unsupervised models
• Neural variational inference
• Continuous/discrete latent variables

Contributions

• Propose and evaluate a range of topic models parameterised with neural networks and trained with variational inference
• Introduce three different neural structures for constructing topic distributions: the Gaussian Softmax distribution (GSM), the Gaussian Stick Breaking distribution (GSB), and the Recurrent Stick Breaking process (RSB)
• Combine the merits of both neural networks and traditional probabilistic topic models
• Interpretable and explicitly represent the dependencies amongst the random variables

Parameterising Topic Distributions

Probabilistic topic models, such as LDA

• Use the latent variables θd and zn for the topic proportion of document d, and the topic assignment for the observed word wn, respectively

Proposed

• Introduce a neural network to parameterise the multinomial topic distribution

The Gaussian Softmax Construction

The Gaussian Stick Breaking Construction

The Recurrent Stick Breaking Construction

Models

Neural Topic Models

Recurrent Neural Topic Models

Topic vs. Document Models

Experiments

Dataset

MXM2 song lyrics, 20NewsGroups3 and Reuters RCV1-v24 news

Evaluation

Document perplexity

Results

Summary

In this paper we have introduced a family of neural topic models using the Gaussian Softmax, Gaussian StickBreaking and Recurrent Stick-Breaking constructions for parameterising the latent multinomial topic distributions of
each document.

Reference

Miao, Yishu, Edward Grefenstette, and Phil Blunsom. “Discovering discrete latent topics with neural variational inference.” Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.