Evaluating Negative Sampling Approaches for Neural Topic Models

• URL: http://arxiv.org/abs/2503.18167v2
• Date: Tue, 25 Mar 2025 05:53:08 GMT
• Title: Evaluating Negative Sampling Approaches for Neural Topic Models
• Authors: Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal, Partha Pratim Das,
• Abstract summary: Negative sampling has emerged as an effective technique that enables deep learning models to learn better representations.<n>We present a comprehensive analysis of the impact of different negative sampling strategies on neural topic models.
• Score: 2.772397333091846
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Negative sampling has emerged as an effective technique that enables deep learning models to learn better representations by introducing the paradigm of learn-to-compare. The goal of this approach is to add robustness to deep learning models to learn better representation by comparing the positive samples against the negative ones. Despite its numerous demonstrations in various areas of computer vision and natural language processing, a comprehensive study of the effect of negative sampling in an unsupervised domain like topic modeling has not been well explored. In this paper, we present a comprehensive analysis of the impact of different negative sampling strategies on neural topic models. We compare the performance of several popular neural topic models by incorporating a negative sampling technique in the decoder of variational autoencoder-based neural topic models. Experiments on four publicly available datasets demonstrate that integrating negative sampling into topic models results in significant enhancements across multiple aspects, including improved topic coherence, richer topic diversity, and more accurate document classification. Manual evaluations also indicate that the inclusion of negative sampling into neural topic models enhances the quality of the generated topics. These findings highlight the potential of negative sampling as a valuable tool for advancing the effectiveness of neural topic models.

Related papers

• Self-Supervised Learning for Neural Topic Models with Variance-Invariance-Covariance Regularization [12.784397404903142]
  We propose a self-supervised neural topic model (NTM) that combines the power of NTMs and regularized self-supervised learning methods to improve performance.<n>NTMs use neural networks to learn latent topics hidden behind the words in documents.<n>Our models outperformed baselines and state-of-the-art models both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2025-02-14T06:47:37Z)
• Rethinking negative sampling in content-based news recommendation [1.5416095780642964]
  News recommender systems are hindered by the brief lifespan of articles, as they undergo rapid relevance decay. Recent studies have demonstrated the potential of content-based neural techniques in tackling this problem. In this study, we posit that the careful sampling of negative examples has a big impact on the model's outcome.
  arXiv  Detail & Related papers  (2024-11-13T15:42:13Z)
• Generating Enhanced Negatives for Training Language-Based Object Detectors [86.1914216335631]
  We propose to leverage the vast knowledge built into modern generative models to automatically build negatives that are more relevant to the original data. Specifically, we use large-language-models to generate negative text descriptions, and text-to-image diffusion models to also generate corresponding negative images. Our experimental analysis confirms the relevance of the generated negative data, and its use in language-based detectors improves performance on two complex benchmarks.
  arXiv  Detail & Related papers  (2023-12-29T23:04:00Z)
• Are Neural Topic Models Broken? [81.15470302729638]
  We study the relationship between automated and human evaluation of topic models. We find that neural topic models fare worse in both respects compared to an established classical method.
  arXiv  Detail & Related papers  (2022-10-28T14:38:50Z)
• A Joint Learning Approach for Semi-supervised Neural Topic Modeling [25.104653662416023]
  We introduce the Label-Indexed Neural Topic Model (LI-NTM), which is the first effective upstream semi-supervised neural topic model. We find that LI-NTM outperforms existing neural topic models in document reconstruction benchmarks.
  arXiv  Detail & Related papers  (2022-04-07T04:42:17Z)
• Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling [8.329870357145927]
  Coherence evaluation of machine generated text is one of the principal applications of coherence models that needs to be investigated. We explore training data and self-supervision objectives that result in a model that generalizes well across tasks. We show empirically that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples.
  arXiv  Detail & Related papers  (2021-10-14T07:44:14Z)
• Rethinking InfoNCE: How Many Negative Samples Do You Need? [54.146208195806636]
  We study how many negative samples are optimal for InfoNCE in different scenarios via a semi-quantitative theoretical framework. We estimate the optimal negative sampling ratio using the $K$ value that maximizes the training effectiveness function.
  arXiv  Detail & Related papers  (2021-05-27T08:38:29Z)
• Have you tried Neural Topic Models? Comparative Analysis of Neural and Non-Neural Topic Models with Application to COVID-19 Twitter Data [11.199249808462458]
  We conduct a comparative study examining state-of-the-art neural versus non-neural topic models. We show that neural topic models outperform their classical counterparts on standard evaluation metrics. We also propose a novel regularization term for neural topic models, which is designed to address the well-documented problem of mode collapse.
  arXiv  Detail & Related papers  (2021-05-21T07:24:09Z)
• Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profiles [69.9674326582747]
  This paper presents a visual framework to investigate neural network models subjected to adversarial examples. We show how observing these elements can quickly pinpoint exploited areas in a model.
  arXiv  Detail & Related papers  (2021-03-18T13:04:21Z)
• On the Transferability of Adversarial Attacksagainst Neural Text Classifier [121.6758865857686]
  We investigate the transferability of adversarial examples for text classification models. We propose a genetic algorithm to find an ensemble of models that can induce adversarial examples to fool almost all existing models. We derive word replacement rules that can be used for model diagnostics from these adversarial examples.
  arXiv  Detail & Related papers  (2020-11-17T10:45:05Z)
• Reinforced Negative Sampling over Knowledge Graph for Recommendation [106.07209348727564]
  We develop a new negative sampling model, Knowledge Graph Policy Network (kgPolicy), which works as a reinforcement learning agent to explore high-quality negatives. kgPolicy navigates from the target positive interaction, adaptively receives knowledge-aware negative signals, and ultimately yields a potential negative item to train the recommender.
  arXiv  Detail & Related papers  (2020-03-12T12:44:30Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.