Disentangled Representations from Non-Disentangled Models
- URL: http://arxiv.org/abs/2102.06204v1
- Date: Thu, 11 Feb 2021 18:59:43 GMT
- Title: Disentangled Representations from Non-Disentangled Models
- Authors: Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, Artem Babenko
- Abstract summary: This paper investigates an alternative route to disentangled representations.
Namely, we propose to extract such representations from the state-of-the-art generative models trained without disangling terms in their objectives.
- Score: 25.272389610447856
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Constructing disentangled representations is known to be a difficult task,
especially in the unsupervised scenario. The dominating paradigm of
unsupervised disentanglement is currently to train a generative model that
separates different factors of variation in its latent space. This separation
is typically enforced by training with specific regularization terms in the
model's objective function. These terms, however, introduce additional
hyperparameters responsible for the trade-off between disentanglement and
generation quality. While tuning these hyperparameters is crucial for proper
disentanglement, it is often unclear how to tune them without external
supervision.
This paper investigates an alternative route to disentangled representations.
Namely, we propose to extract such representations from the state-of-the-art
generative models trained without disentangling terms in their objectives. This
paradigm of post hoc disentanglement employs little or no hyperparameters when
learning representations while achieving results on par with existing
state-of-the-art, as shown by comparison in terms of established
disentanglement metrics, fairness, and the abstract reasoning task. All our
code and models are publicly available.
Related papers
- Sequential Representation Learning via Static-Dynamic Conditional Disentanglement [58.19137637859017]
This paper explores self-supervised disentangled representation learning within sequential data, focusing on separating time-independent and time-varying factors in videos.
We propose a new model that breaks the usual independence assumption between those factors by explicitly accounting for the causal relationship between the static/dynamic variables.
Experiments show that the proposed approach outperforms previous complex state-of-the-art techniques in scenarios where the dynamics of a scene are influenced by its content.
arXiv Detail & Related papers (2024-08-10T17:04:39Z) - Inference-Time Rule Eraser: Fair Recognition via Distilling and Removing Biased Rules [16.85221824455542]
Machine learning models often make predictions based on biased features such as gender, race, and other social attributes.
Traditional approaches to addressing this issue involve retraining or fine-tuning neural networks with fairness-aware optimization objectives.
We introduce the Inference-Time Rule Eraser (Eraser), a novel method designed to address fairness concerns.
arXiv Detail & Related papers (2024-04-07T05:47:41Z) - SCADI: Self-supervised Causal Disentanglement in Latent Variable Models [0.0]
We propose a novel model, SCADI(SElf-supervised CAusal DIsentanglement), that enables the model to discover semantic factors and learn their causal relationships without supervision.
This model combines a masked structural causal model (SCM) with a pseudo-label generator for causal disentanglement, aiming to provide a new direction for self-supervised causal disentanglement models.
arXiv Detail & Related papers (2023-11-11T13:33:43Z) - Meaning Representations from Trajectories in Autoregressive Models [106.63181745054571]
We propose to extract meaning representations from autoregressive language models by considering the distribution of all possible trajectories extending an input text.
This strategy is prompt-free, does not require fine-tuning, and is applicable to any pre-trained autoregressive model.
We empirically show that the representations obtained from large models align well with human annotations, outperform other zero-shot and prompt-free methods on semantic similarity tasks, and can be used to solve more complex entailment and containment tasks that standard embeddings cannot handle.
arXiv Detail & Related papers (2023-10-23T04:35:58Z) - Representation Disentaglement via Regularization by Causal
Identification [3.9160947065896803]
We propose the use of a causal collider structured model to describe the underlying data generative process assumptions in disentangled representation learning.
For this, we propose regularization by identification (ReI), a modular regularization engine designed to align the behavior of large scale generative models with the disentanglement constraints imposed by causal identification.
arXiv Detail & Related papers (2023-02-28T23:18:54Z) - Toward Certified Robustness Against Real-World Distribution Shifts [65.66374339500025]
We train a generative model to learn perturbations from data and define specifications with respect to the output of the learned model.
A unique challenge arising from this setting is that existing verifiers cannot tightly approximate sigmoid activations.
We propose a general meta-algorithm for handling sigmoid activations which leverages classical notions of counter-example-guided abstraction refinement.
arXiv Detail & Related papers (2022-06-08T04:09:13Z) - Unsupervised Disentanglement with Tensor Product Representations on the
Torus [78.6315881294899]
Current methods for learning representations with auto-encoders almost exclusively employ vectors as the latent representations.
In this work, we propose to employ a tensor product structure for this purpose.
In contrast to the conventional variations methods, which are targeted toward normally distributed features, the latent space in our representation is distributed uniformly over a set of unit circles.
arXiv Detail & Related papers (2022-02-13T04:23:12Z) - Do Generative Models Know Disentanglement? Contrastive Learning is All
You Need [59.033559925639075]
We propose an unsupervised and model-agnostic method: Disentanglement via Contrast (DisCo) in the Variation Space.
DisCo achieves the state-of-the-art disentanglement given pretrained non-disentangled generative models, including GAN, VAE, and Flow.
arXiv Detail & Related papers (2021-02-21T08:01:20Z) - Linear Disentangled Representations and Unsupervised Action Estimation [2.793095554369282]
We show that linear disentangled representations are not generally present in standard VAE models.
We propose a method to induce irreducible representations which forgoes the need for labelled action sequences.
arXiv Detail & Related papers (2020-08-18T13:23:57Z) - Evaluating the Disentanglement of Deep Generative Models through
Manifold Topology [66.06153115971732]
We present a method for quantifying disentanglement that only uses the generative model.
We empirically evaluate several state-of-the-art models across multiple datasets.
arXiv Detail & Related papers (2020-06-05T20:54:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.