Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions

• URL: http://arxiv.org/abs/2602.17907v1
• Date: Fri, 20 Feb 2026 00:12:04 GMT
• Title: Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions
• Authors: Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini,
• Abstract summary: We propose a novel approach to construct semantically-grounded soft label targets using Language Models (LMs)<n>Our method produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus.<n>We also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents.
• Score: 15.97570754056266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we propose a novel approach to construct semantically-grounded soft label targets using Language Models (LMs) by projecting the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary to obtain contextually enriched supervision signals. By training the topic models to reconstruct the soft labels using the LM hidden states, our method produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Experiments on three datasets show that our method achieves substantial improvements in topic coherence, purity over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications.

Related papers

• Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings [25.565372681837697]
  We introduce a Multi-Granularity Hard-negative (MGH) synthesis framework to generate diverse negative samples with varying levels of similarity with the query.<n>We also propose an Anchor Token Aware (ATA) pooling method to improve text embedding accuracy without increasing model complexity.
  arXiv  Detail & Related papers  (2025-08-31T13:24:48Z)
• Pointer-Guided Pre-Training: Infusing Large Language Models with Paragraph-Level Contextual Awareness [3.2925222641796554]
  "pointer-guided segment ordering" (SO) is a novel pre-training technique aimed at enhancing the contextual understanding of paragraph-level text representations. Our experiments show that pointer-guided pre-training significantly enhances the model's ability to understand complex document structures.
  arXiv  Detail & Related papers  (2024-06-06T15:17:51Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency [71.42261918225773]
  Conceptually, LOCCO can be viewed as a form of self-learning where the semantic being trained is used to generate annotations for unlabeled text. As an added bonus, the annotations produced by LOCCO can be trivially repurposed to train a neural text generation model.
  arXiv  Detail & Related papers  (2023-05-31T16:47:20Z)
• Topics as Entity Clusters: Entity-based Topics from Large Language Models and Graph Neural Networks [0.6486052012623045]
  We propose a novel topic clustering approach using bimodal vector representations of entities. Our approach is better suited to working with entities in comparison to state-of-the-art models.
  arXiv  Detail & Related papers  (2023-01-06T10:54:54Z)
• Knowledge-Aware Bayesian Deep Topic Model [50.58975785318575]
  We propose a Bayesian generative model for incorporating prior domain knowledge into hierarchical topic modeling. Our proposed model efficiently integrates the prior knowledge and improves both hierarchical topic discovery and document representation.
  arXiv  Detail & Related papers  (2022-09-20T09:16:05Z)
• Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets [56.018551958004814]
  This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources. Large-scale datasets with noisy image-text pairs provide a sub-optimal source of supervision. We propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component.
  arXiv  Detail & Related papers  (2021-11-24T19:00:05Z)
• Transferring Semantic Knowledge Into Language Encoders [6.85316573653194]
  We introduce semantic form mid-tuning, an approach for transferring semantic knowledge from semantic meaning representations into language encoders. We show that this alignment can be learned implicitly via classification or directly via triplet loss. Our method yields language encoders that demonstrate improved predictive performance across inference, reading comprehension, textual similarity, and other semantic tasks.
  arXiv  Detail & Related papers  (2021-10-14T14:11:12Z)
• Obtaining Better Static Word Embeddings Using Contextual Embedding Models [53.86080627007695]
  Our proposed distillation method is a simple extension of CBOW-based training. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings.
  arXiv  Detail & Related papers  (2021-06-08T12:59:32Z)
• TAN-NTM: Topic Attention Networks for Neural Topic Modeling [8.631228373008478]
  We propose a novel framework: TAN-NTM which models document as a sequence of tokens instead of BoW at the input layer. We apply attention on LSTM outputs to empower the model to attend on relevant words which convey topic related cues. TAN-NTM achieves state-of-the-art results with 9-15 percentage improvement over score of existing SOTA topic models in NPMI coherence metric.
  arXiv  Detail & Related papers  (2020-12-02T20:58:04Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.