Context Reinforced Neural Topic Modeling over Short Texts

• URL: http://arxiv.org/abs/2008.04545v1
• Date: Tue, 11 Aug 2020 06:41:53 GMT
• Title: Context Reinforced Neural Topic Modeling over Short Texts
• Authors: Jiachun Feng, Zusheng Zhang, Cheng Ding, Yanghui Rao and Haoran Xie
• Abstract summary: We propose a Context Reinforced Neural Topic Model (CRNTM) CRNTM infers the topic for each word in a narrow range by assuming that each short text covers only a few salient topics. Experiments on two benchmark datasets validate the effectiveness of the proposed model on both topic discovery and text classification.
• Score: 15.487822291146689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As one of the prevalent topic mining tools, neural topic modeling has attracted a lot of interests for the advantages of high efficiency in training and strong generalisation abilities. However, due to the lack of context in each short text, the existing neural topic models may suffer from feature sparsity on such documents. To alleviate this issue, we propose a Context Reinforced Neural Topic Model (CRNTM), whose characteristics can be summarized as follows. Firstly, by assuming that each short text covers only a few salient topics, CRNTM infers the topic for each word in a narrow range. Secondly, our model exploits pre-trained word embeddings by treating topics as multivariate Gaussian distributions or Gaussian mixture distributions in the embedding space. Extensive experiments on two benchmark datasets validate the effectiveness of the proposed model on both topic discovery and text classification.

Related papers

• Let the Pretrained Language Models "Imagine" for Short Texts Topic Modeling [29.87929724277381]
  In short texts, co-occurrence information is minimal, which results in feature sparsity in document representation. Existing topic models (probabilistic or neural) mostly fail to mine patterns from them to generate coherent topics. We extend short text into longer sequences using existing pre-trained language models (PLMs)
  arXiv  Detail & Related papers  (2023-10-24T00:23:30Z)
• Effective Neural Topic Modeling with Embedding Clustering Regularization [21.692088899479934]
  We propose a new neural topic model, Embedding Clustering Regularization Topic Model (ECRTM) ECRTM forces each topic embedding to be the center of a separately aggregated word embedding cluster in the semantic space. Our ECRTM generates diverse and coherent topics together with high-quality topic distributions of documents.
  arXiv  Detail & Related papers  (2023-06-07T07:45:38Z)
• Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics [28.13990734234436]
  Recent work incorporates pre-trained word embeddings into Neural Topic Models (NTMs) In this paper, we conduct thorough experiments showing that directly clustering high-quality sentence embeddings with an appropriate word selecting method can generate more coherent and diverse topics than NTMs.
  arXiv  Detail & Related papers  (2022-04-21T04:26:51Z)
• Author Clustering and Topic Estimation for Short Texts [69.54017251622211]
  We propose a novel model that expands on the Latent Dirichlet Allocation by modeling strong dependence among the words in the same document. We also simultaneously cluster users, removing the need for post-hoc cluster estimation. Our method performs as well as -- or better -- than traditional approaches to problems arising in short text.
  arXiv  Detail & Related papers  (2021-06-15T20:55:55Z)
• Be More with Less: Hypergraph Attention Networks for Inductive Text Classification [56.98218530073927]
  Graph neural networks (GNNs) have received increasing attention in the research community and demonstrated their promising results on this canonical task. Despite the success, their performance could be largely jeopardized in practice since they are unable to capture high-order interaction between words. We propose a principled model -- hypergraph attention networks (HyperGAT) which can obtain more expressive power with less computational consumption for text representation learning.
  arXiv  Detail & Related papers  (2020-11-01T00:21:59Z)
• Improving Neural Topic Models using Knowledge Distillation [84.66983329587073]
  We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality.
  arXiv  Detail & Related papers  (2020-10-05T22:49:16Z)
• Modeling Topical Relevance for Multi-Turn Dialogue Generation [61.87165077442267]
  We propose a new model, named STAR-BTM, to tackle the problem of topic drift in multi-turn dialogue. The Biterm Topic Model is pre-trained on the whole training dataset. Then, the topic level attention weights are computed based on the topic representation of each context. Experimental results on both Chinese customer services data and English Ubuntu dialogue data show that STAR-BTM significantly outperforms several state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-27T03:33:22Z)
• Neural Topic Model via Optimal Transport [24.15046280736009]
  We present a new neural topic model via the theory of optimal transport (OT) Specifically, we propose to learn the topic distribution of a document by directly minimising its OT distance to the document's word distributions. Our proposed model can be trained efficiently with a differentiable loss.
  arXiv  Detail & Related papers  (2020-08-12T06:37:09Z)
• Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling [81.33107307509718]
  We propose a topic adaptive storyteller to model the ability of inter-topic generalization. We also propose a prototype encoding structure to model the ability of intra-topic derivation. Experimental results show that topic adaptation and prototype encoding structure mutually bring benefit to the few-shot model.
  arXiv  Detail & Related papers  (2020-08-11T03:55:11Z)
• Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence [48.765579605145454]
  We propose to explicitly segment target text into fragment units and align them with their data correspondences. The resulting architecture maintains the same expressive power as neural attention models. On both E2E and WebNLG benchmarks, we show the proposed model consistently outperforms its neural attention counterparts.
  arXiv  Detail & Related papers  (2020-05-03T14:28:28Z)
• Neural Topic Modeling with Bidirectional Adversarial Training [37.71988156164695]
  We propose a neural topic modeling approach called Bidirectional Adversarial Topic (BAT) model. BAT builds a two-way projection between the document-topic distribution and the document-word distribution. To incorporate word relatedness information, the Bidirectional Adversarial Topic model with Gaussian (Gaussian-BAT) is extended.
  arXiv  Detail & Related papers  (2020-04-26T09:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.