Neural Dynamic Focused Topic Model
- URL: http://arxiv.org/abs/2301.10988v1
- Date: Thu, 26 Jan 2023 08:37:34 GMT
- Title: Neural Dynamic Focused Topic Model
- Authors: Kostadin Cvejoski, Rams\'es J. S\'anchez, C\'esar Ojeda
- Abstract summary: We leverage recent advances in neural variational inference and present an alternative neural approach to the dynamic Focused Topic Model.
We develop a neural model for topic evolution which exploits sequences of Bernoulli random variables in order to track the appearances of topics.
- Score: 2.9005223064604078
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Topic models and all their variants analyse text by learning meaningful
representations through word co-occurrences. As pointed out by Williamson et
al. (2010), such models implicitly assume that the probability of a topic to be
active and its proportion within each document are positively correlated. This
correlation can be strongly detrimental in the case of documents created over
time, simply because recent documents are likely better described by new and
hence rare topics. In this work we leverage recent advances in neural
variational inference and present an alternative neural approach to the dynamic
Focused Topic Model. Indeed, we develop a neural model for topic evolution
which exploits sequences of Bernoulli random variables in order to track the
appearances of topics, thereby decoupling their activities from their
proportions. We evaluate our model on three different datasets (the UN general
debates, the collection of NeurIPS papers, and the ACL Anthology dataset) and
show that it (i) outperforms state-of-the-art topic models in generalization
tasks and (ii) performs comparably to them on prediction tasks, while employing
roughly the same number of parameters, and converging about two times faster.
Source code to reproduce our experiments is available online.
Related papers
- ANTM: An Aligned Neural Topic Model for Exploring Evolving Topics [1.854328133293073]
This paper presents an algorithmic family of dynamic topic models called Aligned Neural Topic Models (ANTM)
ANTM combines novel data mining algorithms to provide a modular framework for discovering evolving topics.
A Python package is developed for researchers and scientists who wish to study the trends and evolving patterns of topics in large-scale textual data.
arXiv Detail & Related papers (2023-02-03T02:31:12Z) - A Joint Learning Approach for Semi-supervised Neural Topic Modeling [25.104653662416023]
We introduce the Label-Indexed Neural Topic Model (LI-NTM), which is the first effective upstream semi-supervised neural topic model.
We find that LI-NTM outperforms existing neural topic models in document reconstruction benchmarks.
arXiv Detail & Related papers (2022-04-07T04:42:17Z) - Dynamically-Scaled Deep Canonical Correlation Analysis [77.34726150561087]
Canonical Correlation Analysis (CCA) is a method for feature extraction of two views by finding maximally correlated linear projections of them.
We introduce a novel dynamic scaling method for training an input-dependent canonical correlation model.
arXiv Detail & Related papers (2022-03-23T12:52:49Z) - A comprehensive comparative evaluation and analysis of Distributional
Semantic Models [61.41800660636555]
We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT.
The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous.
We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
arXiv Detail & Related papers (2021-05-20T15:18:06Z) - A Discrete Variational Recurrent Topic Model without the
Reparametrization Trick [16.54912614895861]
We show how to learn a neural topic model with discrete random variables.
We show improved perplexity and document understanding across multiple corpora.
arXiv Detail & Related papers (2020-10-22T20:53:44Z) - Improving Neural Topic Models using Knowledge Distillation [84.66983329587073]
We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers.
Our modular method can be straightforwardly applied with any neural topic model to improve topic quality.
arXiv Detail & Related papers (2020-10-05T22:49:16Z) - Context Reinforced Neural Topic Modeling over Short Texts [15.487822291146689]
We propose a Context Reinforced Neural Topic Model (CRNTM)
CRNTM infers the topic for each word in a narrow range by assuming that each short text covers only a few salient topics.
Experiments on two benchmark datasets validate the effectiveness of the proposed model on both topic discovery and text classification.
arXiv Detail & Related papers (2020-08-11T06:41:53Z) - Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling [81.33107307509718]
We propose a topic adaptive storyteller to model the ability of inter-topic generalization.
We also propose a prototype encoding structure to model the ability of intra-topic derivation.
Experimental results show that topic adaptation and prototype encoding structure mutually bring benefit to the few-shot model.
arXiv Detail & Related papers (2020-08-11T03:55:11Z) - Neural Topic Models with Survival Supervision: Jointly Predicting Time-to-Event Outcomes and Learning How Clinical Features Relate [10.709447977149532]
We present a neural network framework for learning a survival model to predict a time-to-event outcome.
In particular, we model each subject as a distribution over "topics"
The presence of a topic in a subject means that specific clinical features are more likely to appear for the subject.
arXiv Detail & Related papers (2020-07-15T16:20:04Z) - Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization.
We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise.
We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
arXiv Detail & Related papers (2020-06-11T17:29:53Z) - Rethinking Generalization of Neural Models: A Named Entity Recognition
Case Study [81.11161697133095]
We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives.
Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models.
As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
arXiv Detail & Related papers (2020-01-12T04:33:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.