Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network

• URL: http://arxiv.org/abs/2107.02757v1
• Date: Wed, 30 Jun 2021 10:14:57 GMT
• Title: Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network
• Authors: Zhibin Duan, Dongsheng Wang, Bo Chen, Chaojie Wang, Wenchao Chen, Yewen Li, Jie Ren, Mingyuan Zhou
• Abstract summary: We propose sawtooth factorial topic embedding guided GBN, a deep generative model of documents. Both the words and topics are represented as embedding vectors of the same dimension. Our models outperform other neural topic models on extracting deeper interpretable topics.
• Score: 49.458250193768826
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hierarchical topic models such as the gamma belief network (GBN) have delivered promising results in mining multi-layer document representations and discovering interpretable topic taxonomies. However, they often assume in the prior that the topics at each layer are independently drawn from the Dirichlet distribution, ignoring the dependencies between the topics both at the same layer and across different layers. To relax this assumption, we propose sawtooth factorial topic embedding guided GBN, a deep generative model of documents that captures the dependencies and semantic similarities between the topics in the embedding space. Specifically, both the words and topics are represented as embedding vectors of the same dimension. The topic matrix at a layer is factorized into the product of a factor loading matrix and a topic embedding matrix, the transpose of which is set as the factor loading matrix of the layer above. Repeating this particular type of factorization, which shares components between adjacent layers, leads to a structure referred to as sawtooth factorization. An auto-encoding variational inference network is constructed to optimize the model parameter via stochastic gradient descent. Experiments on big corpora show that our models outperform other neural topic models on extracting deeper interpretable topics and deriving better document representations.

Related papers

• Quantization of Large Language Models with an Overdetermined Basis [73.79368761182998]
  We introduce an algorithm for data quantization based on the principles of Kashin representation. Our findings demonstrate that Kashin Quantization achieves competitive or superior quality in model performance.
  arXiv  Detail & Related papers  (2024-04-15T12:38:46Z)
• The Geometric Structure of Topic Models [0.0]
  Despite their widespread use in research and application, an in-depth analysis of topic models is still an open research topic. We propose an incidence-geometric method for deriving an ordinal structure from flat topic models. We present a new visualization paradigm for concept hierarchies based on ordinal motifs.
  arXiv  Detail & Related papers  (2024-03-06T10:53:51Z)
• Two to Five Truths in Non-Negative Matrix Factorization [0.0]
  We present a scaling inspired by the normalized Laplacian (NL) for graphs that can greatly improve the quality of a non-negative matrix factorization. The matrix scaling gives significant improvement for text topic models in a variety of datasets.
  arXiv  Detail & Related papers  (2023-05-06T14:40:20Z)
• Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings [46.324584649014284]
  A topic model is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions. This paper introduces a new topic-modeling framework where each document is viewed as a set of word embedding vectors and each topic is modeled as an embedding vector in the same embedding space. Embedding the words and topics in the same vector space, we define a method to measure the semantic difference between the embedding vectors of the words of a document and these of the topics, and optimize the topic embeddings to minimize the expected difference over all documents.
  arXiv  Detail & Related papers  (2022-03-03T08:46:23Z)
• Topic Analysis for Text with Side Data [18.939336393665553]
  We introduce a hybrid generative probabilistic model that combines a neural network with a latent topic model. In the model, each document is modeled as a finite mixture over an underlying set of topics. Each topic is modeled as an infinite mixture over an underlying set of topic probabilities.
  arXiv  Detail & Related papers  (2022-03-01T22:06:30Z)
• Revisiting Over-smoothing in BERT from the Perspective of Graph [111.24636158179908]
  Recently over-smoothing phenomenon of Transformer-based models is observed in both vision and language fields. We find that layer normalization plays a key role in the over-smoothing issue of Transformer-based models. We consider hierarchical fusion strategies, which combine the representations from different layers adaptively to make the output more diverse.
  arXiv  Detail & Related papers  (2022-02-17T12:20:52Z)
• TopicNet: Semantic Graph-Guided Topic Discovery [51.71374479354178]
  Existing deep hierarchical topic models are able to extract semantically meaningful topics from a text corpus in an unsupervised manner. We introduce TopicNet as a deep hierarchical topic model that can inject prior structural knowledge as an inductive bias to influence learning.
  arXiv  Detail & Related papers  (2021-10-27T09:07:14Z)
• EigenGAN: Layer-Wise Eigen-Learning for GANs [84.33920839885619]
  EigenGAN is able to unsupervisedly mine interpretable and controllable dimensions from different generator layers. By traversing the coefficient of a specific eigen-dimension, the generator can produce samples with continuous changes corresponding to a specific semantic attribute.
  arXiv  Detail & Related papers  (2021-04-26T11:14:37Z)
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference [55.35176938713946]
  We develop deep autoencoding topic model (DATM) that uses a hierarchy of gamma distributions to construct its multi-stochastic-layer generative network. We propose a Weibull upward-downward variational encoder that deterministically propagates information upward via a deep neural network, followed by a downward generative model. The efficacy and scalability of our models are demonstrated on both unsupervised and supervised learning tasks on big corpora.
  arXiv  Detail & Related papers  (2020-06-15T22:22:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.