Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling

• URL: http://arxiv.org/abs/2303.15973v1
• Date: Tue, 28 Mar 2023 13:45:39 GMT
• Title: Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling
• Authors: Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal
• Abstract summary: Dropout is a widely used regularization trick to resolve the overfitting issue in large feedforward neural networks trained on a small dataset. We have analyzed the consequences of dropout in the encoder as well as in the decoder of the VAE architecture in three widely used neural topic models.
• Score: 0.6445605125467573
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dropout is a widely used regularization trick to resolve the overfitting issue in large feedforward neural networks trained on a small dataset, which performs poorly on the held-out test subset. Although the effectiveness of this regularization trick has been extensively studied for convolutional neural networks, there is a lack of analysis of it for unsupervised models and in particular, VAE-based neural topic models. In this paper, we have analyzed the consequences of dropout in the encoder as well as in the decoder of the VAE architecture in three widely used neural topic models, namely, contextualized topic model (CTM), ProdLDA, and embedded topic model (ETM) using four publicly available datasets. We characterize the dropout effect on these models in terms of the quality and predictive performance of the generated topics.

Related papers

• Generalized Factor Neural Network Model for High-dimensional Regression [50.554377879576066]
  We tackle the challenges of modeling high-dimensional data sets with latent low-dimensional structures hidden within complex, non-linear, and noisy relationships. Our approach enables a seamless integration of concepts from non-parametric regression, factor models, and neural networks for high-dimensional regression.
  arXiv  Detail & Related papers  (2025-02-16T23:13:55Z)
• Self-Supervised Learning for Neural Topic Models with Variance-Invariance-Covariance Regularization [12.784397404903142]
  We propose a self-supervised neural topic model (NTM) that combines the power of NTMs and regularized self-supervised learning methods to improve performance. NTMs use neural networks to learn latent topics hidden behind the words in documents. Our models outperformed baselines and state-of-the-art models both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2025-02-14T06:47:37Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Application of quantum neural network model to a multivariate regression problem [0.0]
  This study investigates the effect of the size of the training data on generalization performance. The results indicate that QNN is particularly effective when the size of training data is small.
  arXiv  Detail & Related papers  (2023-10-19T08:10:12Z)
• Do We Need an Encoder-Decoder to Model Dynamical Systems on Networks? [18.92828441607381]
  We show that embeddings induce a model that fits observations well but simultaneously has incorrect dynamical behaviours. We propose a simple embedding-free alternative based on parametrising two additive vector-field components.
  arXiv  Detail & Related papers  (2023-05-20T12:41:47Z)
• Pre-training via Denoising for Molecular Property Prediction [53.409242538744444]
  We describe a pre-training technique that utilizes large datasets of 3D molecular structures at equilibrium. Inspired by recent advances in noise regularization, our pre-training objective is based on denoising.
  arXiv  Detail & Related papers  (2022-05-31T22:28:34Z)
• A Joint Learning Approach for Semi-supervised Neural Topic Modeling [25.104653662416023]
  We introduce the Label-Indexed Neural Topic Model (LI-NTM), which is the first effective upstream semi-supervised neural topic model. We find that LI-NTM outperforms existing neural topic models in document reconstruction benchmarks.
  arXiv  Detail & Related papers  (2022-04-07T04:42:17Z)
• EINNs: Epidemiologically-Informed Neural Networks [75.34199997857341]
  We introduce a new class of physics-informed neural networks-EINN-crafted for epidemic forecasting. We investigate how to leverage both the theoretical flexibility provided by mechanistic models as well as the data-driven expressability afforded by AI models.
  arXiv  Detail & Related papers  (2022-02-21T18:59:03Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Have you tried Neural Topic Models? Comparative Analysis of Neural and Non-Neural Topic Models with Application to COVID-19 Twitter Data [11.199249808462458]
  We conduct a comparative study examining state-of-the-art neural versus non-neural topic models. We show that neural topic models outperform their classical counterparts on standard evaluation metrics. We also propose a novel regularization term for neural topic models, which is designed to address the well-documented problem of mode collapse.
  arXiv  Detail & Related papers  (2021-05-21T07:24:09Z)
• Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profiles [69.9674326582747]
  This paper presents a visual framework to investigate neural network models subjected to adversarial examples. We show how observing these elements can quickly pinpoint exploited areas in a model.
  arXiv  Detail & Related papers  (2021-03-18T13:04:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.