Self-Supervised Learning for Neural Topic Models with Variance-Invariance-Covariance Regularization

• URL: http://arxiv.org/abs/2502.09944v1
• Date: Fri, 14 Feb 2025 06:47:37 GMT
• Title: Self-Supervised Learning for Neural Topic Models with Variance-Invariance-Covariance Regularization
• Authors: Weiran Xu, Kengo Hirami, Koji Eguchi,
• Abstract summary: We propose a self-supervised neural topic model (NTM) that combines the power of NTMs and regularized self-supervised learning methods to improve performance. NTMs use neural networks to learn latent topics hidden behind the words in documents. Our models outperformed baselines and state-of-the-art models both quantitatively and qualitatively.
• Score: 12.784397404903142
• License:
• Abstract: In our study, we propose a self-supervised neural topic model (NTM) that combines the power of NTMs and regularized self-supervised learning methods to improve performance. NTMs use neural networks to learn latent topics hidden behind the words in documents, enabling greater flexibility and the ability to estimate more coherent topics compared to traditional topic models. On the other hand, some self-supervised learning methods use a joint embedding architecture with two identical networks that produce similar representations for two augmented versions of the same input. Regularizations are applied to these representations to prevent collapse, which would otherwise result in the networks outputting constant or redundant representations for all inputs. Our model enhances topic quality by explicitly regularizing latent topic representations of anchor and positive samples. We also introduced an adversarial data augmentation method to replace the heuristic sampling method. We further developed several variation models including those on the basis of an NTM that incorporates contrastive learning with both positive and negative samples. Experimental results on three datasets showed that our models outperformed baselines and state-of-the-art models both quantitatively and qualitatively.

Related papers

• Multi-Head Self-Attending Neural Tucker Factorization [5.734615417239977]
  We introduce a neural network-based tensor factorization approach tailored for learning representations of high-dimensional and incomplete (HDI) tensors. The proposed MSNTucF model demonstrates superior performance compared to state-of-the-art benchmark models in estimating missing observations.
  arXiv  Detail & Related papers  (2025-01-16T13:04:15Z)
• Dynamic Post-Hoc Neural Ensemblers [55.15643209328513]
  In this study, we explore employing neural networks as ensemble methods. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions. We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-10-06T15:25:39Z)
• Improving Neural Additive Models with Bayesian Principles [54.29602161803093]
  Neural additive models (NAMs) enhance the transparency of deep neural networks by handling calibrated input features in separate additive sub-networks. We develop Laplace-approximated NAMs (LA-NAMs) which show improved empirical performance on datasets and challenging real-world medical tasks.
  arXiv  Detail & Related papers  (2023-05-26T13:19:15Z)
• Diversity-Aware Coherence Loss for Improving Neural Topic Models [20.98172300869239]
  We propose a novel diversity-aware coherence loss that encourages the model to learn corpus-level coherence scores. Experimental results on multiple datasets show that our method significantly improves the performance of neural topic models.
  arXiv  Detail & Related papers  (2023-05-25T16:01:56Z)
• A Joint Learning Approach for Semi-supervised Neural Topic Modeling [25.104653662416023]
  We introduce the Label-Indexed Neural Topic Model (LI-NTM), which is the first effective upstream semi-supervised neural topic model. We find that LI-NTM outperforms existing neural topic models in document reconstruction benchmarks.
  arXiv  Detail & Related papers  (2022-04-07T04:42:17Z)
• Dynamically-Scaled Deep Canonical Correlation Analysis [77.34726150561087]
  Canonical Correlation Analysis (CCA) is a method for feature extraction of two views by finding maximally correlated linear projections of them. We introduce a novel dynamic scaling method for training an input-dependent canonical correlation model.
  arXiv  Detail & Related papers  (2022-03-23T12:52:49Z)
• Consistency and Monotonicity Regularization for Neural Knowledge Tracing [50.92661409499299]
  Knowledge Tracing (KT) tracking a human's knowledge acquisition is a central component in online learning and AI in Education. We propose three types of novel data augmentation, coined replacement, insertion, and deletion, along with corresponding regularization losses. Extensive experiments on various KT benchmarks show that our regularization scheme consistently improves the model performances.
  arXiv  Detail & Related papers  (2021-05-03T02:36:29Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [54.94763543386523]
  Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors. We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method. Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
  arXiv  Detail & Related papers  (2020-10-25T18:51:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.