Discrete Latent Variable Representations for Low-Resource Text Classification

• URL: http://arxiv.org/abs/2006.06226v1
• Date: Thu, 11 Jun 2020 06:55:13 GMT
• Title: Discrete Latent Variable Representations for Low-Resource Text Classification
• Authors: Shuning Jin, Sam Wiseman, Karl Stratos, Karen Livescu
• Abstract summary: We consider approaches to learning discrete latent variable models for text. We compare the performance of the learned representations as features for low-resource document and sentence classification. An amortized variant of Hard EM performs particularly well in the lowest-resource regimes.
• Score: 47.936293924113855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While much work on deep latent variable models of text uses continuous latent variables, discrete latent variables are interesting because they are more interpretable and typically more space efficient. We consider several approaches to learning discrete latent variable models for text in the case where exact marginalization over these variables is intractable. We compare the performance of the learned representations as features for low-resource document and sentence classification. Our best models outperform the previous best reported results with continuous representations in these low-resource settings, while learning significantly more compressed representations. Interestingly, we find that an amortized variant of Hard EM performs particularly well in the lowest-resource regimes.

Related papers

• LatentExplainer: Explaining Latent Representations in Deep Generative Models with Multi-modal Foundation Models [4.675123839851372]
  textitLatentExplainer is a framework for automatically generating semantically meaningful explanations of latent variables in deep generative models. Our approach perturbs latent variables, interpreting changes in generated data, and uses multi-modal large language models (MLLMs) to produce human-understandable explanations.
  arXiv  Detail & Related papers  (2024-06-21T04:39:03Z)
• Learning Sparsity of Representations with Discrete Latent Variables [15.05207849434673]
  We propose a sparse deep latent generative model SDLGM to explicitly model degree of sparsity. The resulting sparsity of a representation is not fixed, but fits to the observation itself under the pre-defined restriction. For inference and learning, we develop an amortized variational method based on MC gradient estimator.
  arXiv  Detail & Related papers  (2023-04-03T12:47:18Z)
• Learning Context-aware Classifier for Semantic Segmentation [88.88198210948426]
  In this paper, contextual hints are exploited via learning a context-aware classifier. Our method is model-agnostic and can be easily applied to generic segmentation models. With only negligible additional parameters and +2% inference time, decent performance gain has been achieved on both small and large models.
  arXiv  Detail & Related papers  (2023-03-21T07:00:35Z)
• Interpretable Latent Variables in Deep State Space Models [4.884336328409872]
  We introduce a new version of deep state-space models (DSSMs) that combines a recurrent neural network with a state-space framework to forecast time series data. The model estimates the observed series as functions of latent variables that evolve non-linearly through time.
  arXiv  Detail & Related papers  (2022-03-03T23:10:58Z)
• Learning Conditional Invariance through Cycle Consistency [60.85059977904014]
  We propose a novel approach to identify meaningful and independent factors of variation in a dataset. Our method involves two separate latent subspaces for the target property and the remaining input information. We demonstrate on synthetic and molecular data that our approach identifies more meaningful factors which lead to sparser and more interpretable models.
  arXiv  Detail & Related papers  (2021-11-25T17:33:12Z)
• HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization [57.798070356553936]
  HETFORMER is a Transformer-based pre-trained model with multi-granularity sparse attentions for extractive summarization. Experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1.
  arXiv  Detail & Related papers  (2021-10-12T22:42:31Z)
• Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders [0.0]
  We argue that continuous variables may not be ideal to model features of textual data, due to the fact that most generative factors in text are discrete. We propose a Variational Autoencoder based method which models language features as discrete variables and encourages independence between variables for learning disentangled representations.
  arXiv  Detail & Related papers  (2021-09-15T09:10:05Z)
• Discrete Auto-regressive Variational Attention Models for Text Modeling [53.38382932162732]
  Variational autoencoders (VAEs) have been widely applied for text modeling. They are troubled by two challenges: information underrepresentation and posterior collapse. We propose Discrete Auto-regressive Variational Attention Model (DAVAM) to address the challenges.
  arXiv  Detail & Related papers  (2021-06-16T06:36:26Z)
• Variational Information Bottleneck for Effective Low-Resource Fine-Tuning [40.66716433803935]
  We propose to use Variational Information Bottleneck (VIB) to suppress irrelevant features when fine-tuning on low-resource target tasks. We show that our VIB model finds sentence representations that are more robust to biases in natural language inference datasets.
  arXiv  Detail & Related papers  (2021-06-10T03:08:13Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.