Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders

• URL: http://arxiv.org/abs/2109.12679v4
• Date: Mon, 25 Dec 2023 15:08:02 GMT
• Title: Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders
• Authors: Lisa Bonheme and Marek Grzes
• Abstract summary: The ability of Variational Autoencoders to learn disentangled representations has made them appealing for practical applications. Their mean representations, which are generally used for downstream tasks, have recently been shown to be more correlated than their sampled counterpart. We show that passive variables exhibit high correlation scores with other variables in mean representations while being fully uncorrelated in sampled ones.
• Score: 6.68999512375737
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The ability of Variational Autoencoders to learn disentangled representations has made them appealing for practical applications. However, their mean representations, which are generally used for downstream tasks, have recently been shown to be more correlated than their sampled counterpart, on which disentanglement is usually measured. In this paper, we refine this observation through the lens of selective posterior collapse, which states that only a subset of the learned representations, the active variables, is encoding useful information while the rest (the passive variables) is discarded. We first extend the existing definition to multiple data examples and show that active variables are equally disentangled in mean and sampled representations. Based on this extension and the pre-trained models from disentanglement lib, we then isolate the passive variables and show that they are responsible for the discrepancies between mean and sampled representations. Specifically, passive variables exhibit high correlation scores with other variables in mean representations while being fully uncorrelated in sampled ones. We thus conclude that despite what their higher correlation might suggest, mean representations are still good candidates for downstream tasks applications. However, it may be beneficial to remove their passive variables, especially when used with models sensitive to correlated features.

Related papers

• Unsupervised Representation Learning from Sparse Transformation Analysis [79.94858534887801]
  We propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model.
  arXiv  Detail & Related papers  (2024-10-07T23:53:25Z)
• Even Small Correlation and Diversity Shifts Pose Dataset-Bias Issues [19.4921353136871]
  We study two types of distribution shifts: diversity shifts, which occur when test samples exhibit patterns unseen during training, and correlation shifts, which occur when test data present a different correlation between seen invariant and spurious features. We propose an integrated protocol to analyze both types of shifts using datasets where they co-exist in a controllable manner.
  arXiv  Detail & Related papers  (2023-05-09T23:40:23Z)
• Posterior Collapse and Latent Variable Non-identifiability [54.842098835445]
  We propose a class of latent-identifiable variational autoencoders, deep generative models which enforce identifiability without sacrificing flexibility. Across synthetic and real datasets, latent-identifiable variational autoencoders outperform existing methods in mitigating posterior collapse and providing meaningful representations of the data.
  arXiv  Detail & Related papers  (2023-01-02T06:16:56Z)
• Causal Transportability for Visual Recognition [70.13627281087325]
  We show that standard classifiers fail because the association between images and labels is not transportable across settings. We then show that the causal effect, which severs all sources of confounding, remains invariant across domains. This motivates us to develop an algorithm to estimate the causal effect for image classification.
  arXiv  Detail & Related papers  (2022-04-26T15:02:11Z)
• Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets [53.34152466646884]
  In this paper, we show how bringing recent results on equivariant representation learning instantiated on structured spaces together with simple use of classical results on causal inference provides an effective practical solution. We demonstrate how our model allows dealing with more than one nuisance variable under some assumptions and can enable analysis of pooled scientific datasets in scenarios that would otherwise entail removing a large portion of the samples.
  arXiv  Detail & Related papers  (2022-03-29T04:54:06Z)
• Null-sampling for Interpretable and Fair Representations [8.654168514863649]
  We learn invariant representations, in the data domain, to achieve interpretability in algorithmic fairness. By placing the representations into the data domain, the changes made by the model are easily examinable by human auditors.
  arXiv  Detail & Related papers  (2020-08-12T11:49:01Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)
• Discrete Latent Variable Representations for Low-Resource Text Classification [47.936293924113855]
  We consider approaches to learning discrete latent variable models for text. We compare the performance of the learned representations as features for low-resource document and sentence classification. An amortized variant of Hard EM performs particularly well in the lowest-resource regimes.
  arXiv  Detail & Related papers  (2020-06-11T06:55:13Z)
• Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature Normalization [17.829013101192295]
  Extraneous variables are variables that are irrelevant for a certain task, but heavily affect the distribution of the available data. We show that the presence of such variables can degrade the performance of deep-learning models. We show that estimating the feature statistics adaptively during inference, as in instance normalization, addresses this issue.
  arXiv  Detail & Related papers  (2020-02-10T18:47:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.