GloCOM: A Short Text Neural Topic Model via Global Clustering Context

• URL: http://arxiv.org/abs/2412.00525v2
• Date: Thu, 23 Jan 2025 08:47:52 GMT
• Title: GloCOM: A Short Text Neural Topic Model via Global Clustering Context
• Authors: Quang Duc Nguyen, Tung Nguyen, Duc Anh Nguyen, Linh Ngo Van, Sang Dinh, Thien Huu Nguyen,
• Abstract summary: GloCOM is a novel model for constructing global clustering contexts for short documents. It infers both global topic distributions for clustering contexts and local distributions for individual short texts. Our approach outperforms other state-of-the-art models in both topic quality and document representations.
• Score: 29.685615665355396
• License:
• Abstract: Uncovering hidden topics from short texts is challenging for traditional and neural models due to data sparsity, which limits word co-occurrence patterns, and label sparsity, stemming from incomplete reconstruction targets. Although data aggregation offers a potential solution, existing neural topic models often overlook it due to time complexity, poor aggregation quality, and difficulty in inferring topic proportions for individual documents. In this paper, we propose a novel model, GloCOM (Global Clustering COntexts for Topic Models), which addresses these challenges by constructing aggregated global clustering contexts for short documents, leveraging text embeddings from pre-trained language models. GloCOM can infer both global topic distributions for clustering contexts and local distributions for individual short texts. Additionally, the model incorporates these global contexts to augment the reconstruction loss, effectively handling the label sparsity issue. Extensive experiments on short text datasets show that our approach outperforms other state-of-the-art models in both topic quality and document representations.

Related papers

• Bridging Local Details and Global Context in Text-Attributed Graphs [62.522550655068336]
  GraphBridge is a framework that bridges local and global perspectives by leveraging contextual textual information. Our method achieves state-of-theart performance, while our graph-aware token reduction module significantly enhances efficiency and solves scalability issues.
  arXiv  Detail & Related papers  (2024-06-18T13:35:25Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Let the Pretrained Language Models "Imagine" for Short Texts Topic Modeling [29.87929724277381]
  In short texts, co-occurrence information is minimal, which results in feature sparsity in document representation. Existing topic models (probabilistic or neural) mostly fail to mine patterns from them to generate coherent topics. We extend short text into longer sequences using existing pre-trained language models (PLMs)
  arXiv  Detail & Related papers  (2023-10-24T00:23:30Z)
• Effective Neural Topic Modeling with Embedding Clustering Regularization [21.692088899479934]
  We propose a new neural topic model, Embedding Clustering Regularization Topic Model (ECRTM) ECRTM forces each topic embedding to be the center of a separately aggregated word embedding cluster in the semantic space. Our ECRTM generates diverse and coherent topics together with high-quality topic distributions of documents.
  arXiv  Detail & Related papers  (2023-06-07T07:45:38Z)
• GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization [22.053942327552583]
  We propose a graph contrastive topic enhanced language model (GRETEL) for capturing global semantic information. GRETEL integrates the hierarchical transformer encoder and the graph contrastive learning to fuse the semantic information from the global document context and the gold summary. Experimental results on both general domain and biomedical datasets demonstrate that our proposed method outperforms SOTA methods.
  arXiv  Detail & Related papers  (2022-08-21T23:09:29Z)
• Enriching and Controlling Global Semantics for Text Summarization [11.037667460077813]
  Transformer-based models have been proven effective in the abstractive summarization task by creating fluent and informative summaries. We introduce a neural topic model empowered with normalizing flow to capture the global semantics of the document, which are then integrated into the summarization model. Our method outperforms state-of-the-art summarization models on five common text summarization datasets.
  arXiv  Detail & Related papers  (2021-09-22T09:31:50Z)
• Author Clustering and Topic Estimation for Short Texts [69.54017251622211]
  We propose a novel model that expands on the Latent Dirichlet Allocation by modeling strong dependence among the words in the same document. We also simultaneously cluster users, removing the need for post-hoc cluster estimation. Our method performs as well as -- or better -- than traditional approaches to problems arising in short text.
  arXiv  Detail & Related papers  (2021-06-15T20:55:55Z)
• Modeling Topical Relevance for Multi-Turn Dialogue Generation [61.87165077442267]
  We propose a new model, named STAR-BTM, to tackle the problem of topic drift in multi-turn dialogue. The Biterm Topic Model is pre-trained on the whole training dataset. Then, the topic level attention weights are computed based on the topic representation of each context. Experimental results on both Chinese customer services data and English Ubuntu dialogue data show that STAR-BTM significantly outperforms several state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-27T03:33:22Z)
• Context Reinforced Neural Topic Modeling over Short Texts [15.487822291146689]
  We propose a Context Reinforced Neural Topic Model (CRNTM) CRNTM infers the topic for each word in a narrow range by assuming that each short text covers only a few salient topics. Experiments on two benchmark datasets validate the effectiveness of the proposed model on both topic discovery and text classification.
  arXiv  Detail & Related papers  (2020-08-11T06:41:53Z)
• Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence [48.765579605145454]
  We propose to explicitly segment target text into fragment units and align them with their data correspondences. The resulting architecture maintains the same expressive power as neural attention models. On both E2E and WebNLG benchmarks, we show the proposed model consistently outperforms its neural attention counterparts.
  arXiv  Detail & Related papers  (2020-05-03T14:28:28Z)
• Towards Making the Most of Context in Neural Machine Translation [112.9845226123306]
  We argue that previous research did not make a clear use of the global context. We propose a new document-level NMT framework that deliberately models the local context of each sentence.
  arXiv  Detail & Related papers  (2020-02-19T03:30:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.