Disentangled Representations from Non-Disentangled Models

• URL: http://arxiv.org/abs/2102.06204v1
• Date: Thu, 11 Feb 2021 18:59:43 GMT
• Title: Disentangled Representations from Non-Disentangled Models
• Authors: Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, Artem Babenko
• Abstract summary: This paper investigates an alternative route to disentangled representations. Namely, we propose to extract such representations from the state-of-the-art generative models trained without disangling terms in their objectives.
• Score: 25.272389610447856
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Constructing disentangled representations is known to be a difficult task, especially in the unsupervised scenario. The dominating paradigm of unsupervised disentanglement is currently to train a generative model that separates different factors of variation in its latent space. This separation is typically enforced by training with specific regularization terms in the model's objective function. These terms, however, introduce additional hyperparameters responsible for the trade-off between disentanglement and generation quality. While tuning these hyperparameters is crucial for proper disentanglement, it is often unclear how to tune them without external supervision. This paper investigates an alternative route to disentangled representations. Namely, we propose to extract such representations from the state-of-the-art generative models trained without disentangling terms in their objectives. This paradigm of post hoc disentanglement employs little or no hyperparameters when learning representations while achieving results on par with existing state-of-the-art, as shown by comparison in terms of established disentanglement metrics, fairness, and the abstract reasoning task. All our code and models are publicly available.

Related papers

• Counterfactual Generation from Language Models [64.55296662926919]
  We show that counterfactual reasoning is conceptually distinct from interventions. We propose a framework for generating true string counterfactuals. Our experiments demonstrate that the approach produces meaningful counterfactuals.
  arXiv  Detail & Related papers  (2024-11-11T17:57:30Z)
• Inference-Time Rule Eraser: Fair Recognition via Distilling and Removing Biased Rules [16.85221824455542]
  Machine learning models often make predictions based on biased features such as gender, race, and other social attributes. Traditional approaches to addressing this issue involve retraining or fine-tuning neural networks with fairness-aware optimization objectives. We introduce the Inference-Time Rule Eraser (Eraser), a novel method designed to address fairness concerns.
  arXiv  Detail & Related papers  (2024-04-07T05:47:41Z)
• SCADI: Self-supervised Causal Disentanglement in Latent Variable Models [0.0]
  We propose a novel model, SCADI(SElf-supervised CAusal DIsentanglement), that enables the model to discover semantic factors and learn their causal relationships without supervision. This model combines a masked structural causal model (SCM) with a pseudo-label generator for causal disentanglement, aiming to provide a new direction for self-supervised causal disentanglement models.
  arXiv  Detail & Related papers  (2023-11-11T13:33:43Z)
• Meaning Representations from Trajectories in Autoregressive Models [106.63181745054571]
  We propose to extract meaning representations from autoregressive language models by considering the distribution of all possible trajectories extending an input text. This strategy is prompt-free, does not require fine-tuning, and is applicable to any pre-trained autoregressive model. We empirically show that the representations obtained from large models align well with human annotations, outperform other zero-shot and prompt-free methods on semantic similarity tasks, and can be used to solve more complex entailment and containment tasks that standard embeddings cannot handle.
  arXiv  Detail & Related papers  (2023-10-23T04:35:58Z)
• Representation Disentaglement via Regularization by Causal Identification [3.9160947065896803]
  We propose the use of a causal collider structured model to describe the underlying data generative process assumptions in disentangled representation learning. For this, we propose regularization by identification (ReI), a modular regularization engine designed to align the behavior of large scale generative models with the disentanglement constraints imposed by causal identification.
  arXiv  Detail & Related papers  (2023-02-28T23:18:54Z)
• Toward Certified Robustness Against Real-World Distribution Shifts [65.66374339500025]
  We train a generative model to learn perturbations from data and define specifications with respect to the output of the learned model. A unique challenge arising from this setting is that existing verifiers cannot tightly approximate sigmoid activations. We propose a general meta-algorithm for handling sigmoid activations which leverages classical notions of counter-example-guided abstraction refinement.
  arXiv  Detail & Related papers  (2022-06-08T04:09:13Z)
• Unsupervised Disentanglement with Tensor Product Representations on the Torus [78.6315881294899]
  Current methods for learning representations with auto-encoders almost exclusively employ vectors as the latent representations. In this work, we propose to employ a tensor product structure for this purpose. In contrast to the conventional variations methods, which are targeted toward normally distributed features, the latent space in our representation is distributed uniformly over a set of unit circles.
  arXiv  Detail & Related papers  (2022-02-13T04:23:12Z)
• Do Generative Models Know Disentanglement? Contrastive Learning is All You Need [59.033559925639075]
  We propose an unsupervised and model-agnostic method: Disentanglement via Contrast (DisCo) in the Variation Space. DisCo achieves the state-of-the-art disentanglement given pretrained non-disentangled generative models, including GAN, VAE, and Flow.
  arXiv  Detail & Related papers  (2021-02-21T08:01:20Z)
• Linear Disentangled Representations and Unsupervised Action Estimation [2.793095554369282]
  We show that linear disentangled representations are not generally present in standard VAE models. We propose a method to induce irreducible representations which forgoes the need for labelled action sequences.
  arXiv  Detail & Related papers  (2020-08-18T13:23:57Z)
• Evaluating the Disentanglement of Deep Generative Models through Manifold Topology [66.06153115971732]
  We present a method for quantifying disentanglement that only uses the generative model. We empirically evaluate several state-of-the-art models across multiple datasets.
  arXiv  Detail & Related papers  (2020-06-05T20:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.