Neural Dynamic Focused Topic Model

• URL: http://arxiv.org/abs/2301.10988v1
• Date: Thu, 26 Jan 2023 08:37:34 GMT
• Title: Neural Dynamic Focused Topic Model
• Authors: Kostadin Cvejoski, Rams\'es J. S\'anchez, C\'esar Ojeda
• Abstract summary: We leverage recent advances in neural variational inference and present an alternative neural approach to the dynamic Focused Topic Model. We develop a neural model for topic evolution which exploits sequences of Bernoulli random variables in order to track the appearances of topics.
• Score: 2.9005223064604078
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Topic models and all their variants analyse text by learning meaningful representations through word co-occurrences. As pointed out by Williamson et al. (2010), such models implicitly assume that the probability of a topic to be active and its proportion within each document are positively correlated. This correlation can be strongly detrimental in the case of documents created over time, simply because recent documents are likely better described by new and hence rare topics. In this work we leverage recent advances in neural variational inference and present an alternative neural approach to the dynamic Focused Topic Model. Indeed, we develop a neural model for topic evolution which exploits sequences of Bernoulli random variables in order to track the appearances of topics, thereby decoupling their activities from their proportions. We evaluate our model on three different datasets (the UN general debates, the collection of NeurIPS papers, and the ACL Anthology dataset) and show that it (i) outperforms state-of-the-art topic models in generalization tasks and (ii) performs comparably to them on prediction tasks, while employing roughly the same number of parameters, and converging about two times faster. Source code to reproduce our experiments is available online.

Related papers

• Embedded Topic Models Enhanced by Wikification [3.082729239227955]
  We incorporate the Wikipedia knowledge into a neural topic model to make it aware of named entities. Our experiments show that our method improves the performance of neural topic models in generalizability.
  arXiv  Detail & Related papers  (2024-10-03T12:39:14Z)
• Iterative Improvement of an Additively Regularized Topic Model [0.0]
  We present a method for iterative training of a topic model. Experiments conducted on several collections of natural language texts show that the proposed ITAR model performs better than other popular topic models.
  arXiv  Detail & Related papers  (2024-08-11T18:22:12Z)
• ANTM: An Aligned Neural Topic Model for Exploring Evolving Topics [1.854328133293073]
  This paper presents an algorithmic family of dynamic topic models called Aligned Neural Topic Models (ANTM) ANTM combines novel data mining algorithms to provide a modular framework for discovering evolving topics. A Python package is developed for researchers and scientists who wish to study the trends and evolving patterns of topics in large-scale textual data.
  arXiv  Detail & Related papers  (2023-02-03T02:31:12Z)
• A Joint Learning Approach for Semi-supervised Neural Topic Modeling [25.104653662416023]
  We introduce the Label-Indexed Neural Topic Model (LI-NTM), which is the first effective upstream semi-supervised neural topic model. We find that LI-NTM outperforms existing neural topic models in document reconstruction benchmarks.
  arXiv  Detail & Related papers  (2022-04-07T04:42:17Z)
• Dynamically-Scaled Deep Canonical Correlation Analysis [77.34726150561087]
  Canonical Correlation Analysis (CCA) is a method for feature extraction of two views by finding maximally correlated linear projections of them. We introduce a novel dynamic scaling method for training an input-dependent canonical correlation model.
  arXiv  Detail & Related papers  (2022-03-23T12:52:49Z)
• A comprehensive comparative evaluation and analysis of Distributional Semantic Models [61.41800660636555]
  We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous. We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
  arXiv  Detail & Related papers  (2021-05-20T15:18:06Z)
• A Discrete Variational Recurrent Topic Model without the Reparametrization Trick [16.54912614895861]
  We show how to learn a neural topic model with discrete random variables. We show improved perplexity and document understanding across multiple corpora.
  arXiv  Detail & Related papers  (2020-10-22T20:53:44Z)
• Improving Neural Topic Models using Knowledge Distillation [84.66983329587073]
  We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality.
  arXiv  Detail & Related papers  (2020-10-05T22:49:16Z)
• Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling [81.33107307509718]
  We propose a topic adaptive storyteller to model the ability of inter-topic generalization. We also propose a prototype encoding structure to model the ability of intra-topic derivation. Experimental results show that topic adaptation and prototype encoding structure mutually bring benefit to the few-shot model.
  arXiv  Detail & Related papers  (2020-08-11T03:55:11Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.