Improving Neural Topic Models using Knowledge Distillation
- URL: http://arxiv.org/abs/2010.02377v1
- Date: Mon, 5 Oct 2020 22:49:16 GMT
- Title: Improving Neural Topic Models using Knowledge Distillation
- Authors: Alexander Hoyle, Pranav Goel, Philip Resnik
- Abstract summary: We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers.
Our modular method can be straightforwardly applied with any neural topic model to improve topic quality.
- Score: 84.66983329587073
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Topic models are often used to identify human-interpretable topics to help
make sense of large document collections. We use knowledge distillation to
combine the best attributes of probabilistic topic models and pretrained
transformers. Our modular method can be straightforwardly applied with any
neural topic model to improve topic quality, which we demonstrate using two
models having disparate architectures, obtaining state-of-the-art topic
coherence. We show that our adaptable framework not only improves performance
in the aggregate over all estimated topics, as is commonly reported, but also
in head-to-head comparisons of aligned topics.
Related papers
- Historia Magistra Vitae: Dynamic Topic Modeling of Roman Literature using Neural Embeddings [10.095706051685665]
We compare topic models built using traditional statistical models (LDA and NMF) and the BERT-based model.
We find that while quantitative metrics prefer statistical models, qualitative evaluation finds better insights from the neural model.
arXiv Detail & Related papers (2024-06-27T05:38:49Z) - FASTopic: A Fast, Adaptive, Stable, and Transferable Topic Modeling Paradigm [76.509837704596]
We present FASTopic, a fast, adaptive, stable, and transferable topic model.
We also propose a novel Embedding Transport Plan (ETP) method.
arXiv Detail & Related papers (2024-05-28T09:06:38Z) - GINopic: Topic Modeling with Graph Isomorphism Network [0.8962460460173959]
We introduce GINopic, a topic modeling framework based on graph isomorphism networks to capture the correlation between words.
We demonstrate the effectiveness of GINopic compared to existing topic models and highlight its potential for advancing topic modeling.
arXiv Detail & Related papers (2024-04-02T17:18:48Z) - Improving the TENOR of Labeling: Re-evaluating Topic Models for Content
Analysis [5.757610495733924]
We conduct the first evaluation of neural, supervised and classical topic models in an interactive task based setting.
We show that current automated metrics do not provide a complete picture of topic modeling capabilities.
arXiv Detail & Related papers (2024-01-29T17:54:04Z) - Improving Neural Topic Models with Wasserstein Knowledge Distillation [0.8962460460173959]
We propose a knowledge distillation framework to compress a contextualized topic model without loss in topic quality.
Experiments show that the student trained with knowledge distillation achieves topic coherence much higher than that of the original student model.
arXiv Detail & Related papers (2023-03-27T16:07:44Z) - Are Neural Topic Models Broken? [81.15470302729638]
We study the relationship between automated and human evaluation of topic models.
We find that neural topic models fare worse in both respects compared to an established classical method.
arXiv Detail & Related papers (2022-10-28T14:38:50Z) - Knowledge-Aware Bayesian Deep Topic Model [50.58975785318575]
We propose a Bayesian generative model for incorporating prior domain knowledge into hierarchical topic modeling.
Our proposed model efficiently integrates the prior knowledge and improves both hierarchical topic discovery and document representation.
arXiv Detail & Related papers (2022-09-20T09:16:05Z) - Exploring Strategies for Generalizable Commonsense Reasoning with
Pre-trained Models [62.28551903638434]
We measure the impact of three different adaptation methods on the generalization and accuracy of models.
Experiments with two models show that fine-tuning performs best, by learning both the content and the structure of the task, but suffers from overfitting and limited generalization to novel answers.
We observe that alternative adaptation methods like prefix-tuning have comparable accuracy, but generalize better to unseen answers and are more robust to adversarial splits.
arXiv Detail & Related papers (2021-09-07T03:13:06Z) - Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling [81.33107307509718]
We propose a topic adaptive storyteller to model the ability of inter-topic generalization.
We also propose a prototype encoding structure to model the ability of intra-topic derivation.
Experimental results show that topic adaptation and prototype encoding structure mutually bring benefit to the few-shot model.
arXiv Detail & Related papers (2020-08-11T03:55:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.