A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation

• URL: http://arxiv.org/abs/2010.14766v1
• Date: Tue, 27 Oct 2020 10:17:15 GMT
• Title: A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
• Authors: Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar R\"atsch, Sylvain Gelly, Bernhard Sch\"olkopf, Olivier Bachem
• Abstract summary: We show that the unsupervised learning of disentangled representations is impossible without inductive biases on both the models and the data. We observe that while the different methods successfully enforce properties "encouraged" by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision.
• Score: 63.042651834453544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The idea behind the \emph{unsupervised} learning of \emph{disentangled} representations is that real-world data is generated by a few explanatory factors of variation which can be recovered by unsupervised learning algorithms. In this paper, we provide a sober look at recent progress in the field and challenge some common assumptions. We first theoretically show that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data. Then, we train over $14000$ models covering most prominent methods and evaluation metrics in a reproducible large-scale experimental study on eight data sets. We observe that while the different methods successfully enforce properties "encouraged" by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Furthermore, different evaluation metrics do not always agree on what should be considered "disentangled" and exhibit systematic differences in the estimation. Finally, increased disentanglement does not seem to necessarily lead to a decreased sample complexity of learning for downstream tasks. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision, investigate concrete benefits of enforcing disentanglement of the learned representations, and consider a reproducible experimental setup covering several data sets.

Related papers

• Unsupervised Representation Learning for Time Series: A Review [20.00853543048447]
  Unsupervised representation learning approaches aim to learn discriminative feature representations from unlabeled data, without the requirement of annotating every sample. We conduct a literature review of existing rapidly evolving unsupervised representation learning approaches for time series. We empirically evaluate state-of-the-art approaches, especially the rapidly evolving contrastive learning methods, on 9 diverse real-world datasets.
  arXiv  Detail & Related papers  (2023-08-03T07:28:06Z)
• Unsupervised Few-shot Learning via Deep Laplacian Eigenmaps [13.6555672824229]
  We present an unsupervised few-shot learning method via deep Laplacian eigenmaps. Our method learns representation from unlabeled data by grouping similar samples together. We analytically show how deep Laplacian eigenmaps avoid collapsed representation in unsupervised learning.
  arXiv  Detail & Related papers  (2022-10-07T14:53:03Z)
• An Empirical Study on Disentanglement of Negative-free Contrastive Learning [9.52192693120765]
  We take different negative-free contrastive learning methods to study the disentanglement property of this genre of self-supervised methods. We propose a new disentanglement metric based on Mutual Information between representation and data factors. Our study shows that the investigated methods can learn a well-disentangled subset of representation.
  arXiv  Detail & Related papers  (2022-06-09T20:25:34Z)
• Agree to Disagree: Diversity through Disagreement for Better Transferability [54.308327969778155]
  We propose D-BAT (Diversity-By-disAgreement Training), which enforces agreement among the models on the training data. We show how D-BAT naturally emerges from the notion of generalized discrepancy.
  arXiv  Detail & Related papers  (2022-02-09T12:03:02Z)
• Desiderata for Representation Learning: A Causal Perspective [104.3711759578494]
  We take a causal perspective on representation learning, formalizing non-spuriousness and efficiency (in supervised representation learning) and disentanglement (in unsupervised representation learning) This yields computable metrics that can be used to assess the degree to which representations satisfy the desiderata of interest and learn non-spurious and disentangled representations from single observational datasets.
  arXiv  Detail & Related papers  (2021-09-08T17:33:54Z)
• Unsupervised Embedding Learning from Uncertainty Momentum Modeling [37.674449317054716]
  We propose a novel solution to explicitly model and explore the uncertainty of the given unlabeled learning samples. We leverage such uncertainty modeling momentum to the learning which is helpful to tackle the outliers.
  arXiv  Detail & Related papers  (2021-07-19T14:06:19Z)
• Disambiguation of weak supervision with exponential convergence rates [88.99819200562784]
  In supervised learning, data are annotated with incomplete yet discriminative information. In this paper, we focus on partial labelling, an instance of weak supervision where, from a given input, we are given a set of potential targets. We propose an empirical disambiguation algorithm to recover full supervision from weak supervision.
  arXiv  Detail & Related papers  (2021-02-04T18:14:32Z)
• Odd-One-Out Representation Learning [1.6822770693792826]
  We show that a weakly-supervised downstream task based on odd-one-out observations is suitable for model selection. We also show that a bespoke metric-learning VAE model which performs highly on this task also out-performs other standard unsupervised and a weakly-supervised disentanglement model.
  arXiv  Detail & Related papers  (2020-12-14T22:01:15Z)
• A Commentary on the Unsupervised Learning of Disentangled Representations [63.042651834453544]
  The goal of the unsupervised learning of disentangled representations is to separate the independent explanatory factors of variation in the data without access to supervision. We discuss the theoretical result showing that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases.
  arXiv  Detail & Related papers  (2020-07-28T13:13:45Z)
• Weakly-Supervised Disentanglement Without Compromises [53.55580957483103]
  Intelligent agents should be able to learn useful representations by observing changes in their environment. We model such observations as pairs of non-i.i.d. images sharing at least one of the underlying factors of variation. We show that only knowing how many factors have changed, but not which ones, is sufficient to learn disentangled representations.
  arXiv  Detail & Related papers  (2020-02-07T16:39:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.