Neural Topic Modeling with Deep Mutual Information Estimation

• URL: http://arxiv.org/abs/2203.06298v1
• Date: Sat, 12 Mar 2022 01:08:10 GMT
• Title: Neural Topic Modeling with Deep Mutual Information Estimation
• Authors: Kang Xu and Xiaoqiu Lu and Yuan-fang Li and Tongtong Wu and Guilin Qi and Ning Ye and Dong Wang and Zheng Zhou
• Abstract summary: We propose a neural topic model which incorporates deep mutual information estimation. NTM-DMIE is a neural network method for topic learning. We evaluate NTM-DMIE on several metrics, including accuracy of text clustering, with topic representation, topic uniqueness and topic coherence.
• Score: 23.474848535821994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The emerging neural topic models make topic modeling more easily adaptable and extendable in unsupervised text mining. However, the existing neural topic models is difficult to retain representative information of the documents within the learnt topic representation. In this paper, we propose a neural topic model which incorporates deep mutual information estimation, i.e., Neural Topic Modeling with Deep Mutual Information Estimation(NTM-DMIE). NTM-DMIE is a neural network method for topic learning which maximizes the mutual information between the input documents and their latent topic representation. To learn robust topic representation, we incorporate the discriminator to discriminate negative examples and positive examples via adversarial learning. Moreover, we use both global and local mutual information to preserve the rich information of the input documents in the topic representation. We evaluate NTM-DMIE on several metrics, including accuracy of text clustering, with topic representation, topic uniqueness and topic coherence. Compared to the existing methods, the experimental results show that NTM-DMIE can outperform in all the metrics on the four datasets.

Related papers

• NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization [29.685615665355396]
  We propose a novel framework called NeuroMax for neural topic models. NeuroMax maximizes the mutual information between the topic representation obtained from the encoder in neural topic models and the representation derived from the PLM. Experimental results indicate that NeuroMax reduces inference time, generates more coherent topics and topic groups, and produces more representative document embeddings.
  arXiv  Detail & Related papers  (2024-09-29T15:59:36Z)
• Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis [5.757610495733924]
  We conduct the first evaluation of neural, supervised and classical topic models in an interactive task based setting. We show that current automated metrics do not provide a complete picture of topic modeling capabilities.
  arXiv  Detail & Related papers  (2024-01-29T17:54:04Z)
• A Survey on Neural Topic Models: Methods, Applications, and Challenges [32.510888679613004]
  Topic models have been prevalent for decades to discover latent topics and infer topic proportions of documents in an unsupervised fashion. The rise of neural networks has facilitated the emergence of a new research field -- Neural Topic Models (NTMs) In this paper, we present a comprehensive survey on neural topic models concerning methods, applications, and challenges.
  arXiv  Detail & Related papers  (2024-01-27T08:52:19Z)
• Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling [96.75821232222201]
  Existing research on multimodal relation extraction (MRE) faces two co-existing challenges, internal-information over-utilization and external-information under-exploitation. We propose a novel framework that simultaneously implements the idea of internal-information screening and external-information exploiting.
  arXiv  Detail & Related papers  (2023-05-19T14:56:57Z)
• Experimental Observations of the Topology of Convolutional Neural Network Activations [2.4235626091331737]
  Topological data analysis provides compact, noise-robust representations of complex structures. Deep neural networks (DNNs) learn millions of parameters associated with a series of transformations defined by the model architecture. In this paper, we apply cutting edge techniques from TDA with the goal of gaining insight into the interpretability of convolutional neural networks used for image classification.
  arXiv  Detail & Related papers  (2022-12-01T02:05:44Z)
• Knowledge-Aware Bayesian Deep Topic Model [50.58975785318575]
  We propose a Bayesian generative model for incorporating prior domain knowledge into hierarchical topic modeling. Our proposed model efficiently integrates the prior knowledge and improves both hierarchical topic discovery and document representation.
  arXiv  Detail & Related papers  (2022-09-20T09:16:05Z)
• A Joint Learning Approach for Semi-supervised Neural Topic Modeling [25.104653662416023]
  We introduce the Label-Indexed Neural Topic Model (LI-NTM), which is the first effective upstream semi-supervised neural topic model. We find that LI-NTM outperforms existing neural topic models in document reconstruction benchmarks.
  arXiv  Detail & Related papers  (2022-04-07T04:42:17Z)
• Inter-layer Information Similarity Assessment of Deep Neural Networks Via Topological Similarity and Persistence Analysis of Data Neighbour Dynamics [93.4221402881609]
  The quantitative analysis of information structure through a deep neural network (DNN) can unveil new insights into the theoretical performance of DNN architectures. Inspired by both LS and ID strategies for quantitative information structure analysis, we introduce two novel complimentary methods for inter-layer information similarity assessment. We demonstrate their efficacy in this study by performing analysis on a deep convolutional neural network architecture on image data.
  arXiv  Detail & Related papers  (2020-12-07T15:34:58Z)
• Improving Neural Topic Models using Knowledge Distillation [84.66983329587073]
  We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality.
  arXiv  Detail & Related papers  (2020-10-05T22:49:16Z)
• Modeling Topical Relevance for Multi-Turn Dialogue Generation [61.87165077442267]
  We propose a new model, named STAR-BTM, to tackle the problem of topic drift in multi-turn dialogue. The Biterm Topic Model is pre-trained on the whole training dataset. Then, the topic level attention weights are computed based on the topic representation of each context. Experimental results on both Chinese customer services data and English Ubuntu dialogue data show that STAR-BTM significantly outperforms several state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-27T03:33:22Z)
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference [55.35176938713946]
  We develop deep autoencoding topic model (DATM) that uses a hierarchy of gamma distributions to construct its multi-stochastic-layer generative network. We propose a Weibull upward-downward variational encoder that deterministically propagates information upward via a deep neural network, followed by a downward generative model. The efficacy and scalability of our models are demonstrated on both unsupervised and supervised learning tasks on big corpora.
  arXiv  Detail & Related papers  (2020-06-15T22:22:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.