Linear Disentangled Representations and Unsupervised Action Estimation
- URL: http://arxiv.org/abs/2008.07922v2
- Date: Tue, 15 Dec 2020 14:41:48 GMT
- Title: Linear Disentangled Representations and Unsupervised Action Estimation
- Authors: Matthew Painter, Jonathon Hare and Adam Prugel-Bennett
- Abstract summary: We show that linear disentangled representations are not generally present in standard VAE models.
We propose a method to induce irreducible representations which forgoes the need for labelled action sequences.
- Score: 2.793095554369282
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Disentangled representation learning has seen a surge in interest over recent
times, generally focusing on new models which optimise one of many disparate
disentanglement metrics. Symmetry Based Disentangled Representation learning
introduced a robust mathematical framework that defined precisely what is meant
by a "linear disentangled representation". This framework determined that such
representations would depend on a particular decomposition of the symmetry
group acting on the data, showing that actions would manifest through
irreducible group representations acting on independent representational
subspaces. Caselles-Dupre et al [2019] subsequently proposed the first model to
induce and demonstrate a linear disentangled representation in a VAE model. In
this work we empirically show that linear disentangled representations are not
generally present in standard VAE models and that they instead require altering
the loss landscape to induce them. We proceed to show that such representations
are a desirable property with regard to classical disentanglement metrics.
Finally we propose a method to induce irreducible representations which forgoes
the need for labelled action sequences, as was required by prior work. We
explore a number of properties of this method, including the ability to learn
from action sequences without knowledge of intermediate states and robustness
under visual noise. We also demonstrate that it can successfully learn 4
independent symmetries directly from pixels.
Related papers
- On the Origins of Linear Representations in Large Language Models [51.88404605700344]
We introduce a simple latent variable model to formalize the concept dynamics of the next token prediction.
Experiments show that linear representations emerge when learning from data matching the latent variable model.
We additionally confirm some predictions of the theory using the LLaMA-2 large language model.
arXiv Detail & Related papers (2024-03-06T17:17:36Z) - A U-turn on Double Descent: Rethinking Parameter Counting in Statistical
Learning [68.76846801719095]
We show that double descent appears exactly when and where it occurs, and that its location is not inherently tied to the threshold p=n.
This provides a resolution to tensions between double descent and statistical intuition.
arXiv Detail & Related papers (2023-10-29T12:05:39Z) - Learning Disentangled Discrete Representations [22.5004558029479]
We show the relationship between discrete latent spaces and disentangled representations by replacing the standard Gaussian variational autoencoder with a tailored categorical variational autoencoder.
We provide both analytical and empirical findings that demonstrate the advantages of discrete VAEs for learning disentangled representations.
arXiv Detail & Related papers (2023-07-26T12:29:58Z) - Learning nonparametric latent causal graphs with unknown interventions [18.6470340274888]
We establish conditions under which latent causal graphs are nonparametrically identifiable.
We do not assume the number of hidden variables is known, and we show that at most one unknown intervention per hidden variable is needed.
arXiv Detail & Related papers (2023-06-05T14:06:35Z) - Log-linear Guardedness and its Implications [116.87322784046926]
Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful.
This work formally defines the notion of log-linear guardedness as the inability of an adversary to predict the concept directly from the representation.
We show that, in the binary case, under certain assumptions, a downstream log-linear model cannot recover the erased concept.
arXiv Detail & Related papers (2022-10-18T17:30:02Z) - Efficient Iterative Amortized Inference for Learning Symmetric and
Disentangled Multi-Object Representations [8.163697683448811]
We introduce EfficientMORL, an efficient framework for the unsupervised learning of object-centric representations.
We show that optimization challenges caused by requiring both symmetry and disentanglement can be addressed by high-cost iterative amortized inference.
We demonstrate strong object decomposition and disentanglement on the standard multi-object benchmark while achieving nearly an order of magnitude faster training and test time inference.
arXiv Detail & Related papers (2021-06-07T14:02:49Z) - Disentangled Representations from Non-Disentangled Models [25.272389610447856]
This paper investigates an alternative route to disentangled representations.
Namely, we propose to extract such representations from the state-of-the-art generative models trained without disangling terms in their objectives.
arXiv Detail & Related papers (2021-02-11T18:59:43Z) - Joint Estimation of Image Representations and their Lie Invariants [57.3768308075675]
Images encode both the state of the world and its content.
The automatic extraction of this information is challenging because of the high-dimensionality and entangled encoding inherent to the image representation.
This article introduces two theoretical approaches aimed at the resolution of these challenges.
arXiv Detail & Related papers (2020-12-05T00:07:41Z) - Learning Disentangled Representations with Latent Variation
Predictability [102.4163768995288]
This paper defines the variation predictability of latent disentangled representations.
Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs.
We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
arXiv Detail & Related papers (2020-07-25T08:54:26Z) - Deep Dimension Reduction for Supervised Representation Learning [51.10448064423656]
We propose a deep dimension reduction approach to learning representations with essential characteristics.
The proposed approach is a nonparametric generalization of the sufficient dimension reduction method.
We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero.
arXiv Detail & Related papers (2020-06-10T14:47:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.