Odd-One-Out Representation Learning

• URL: http://arxiv.org/abs/2012.07966v1
• Date: Mon, 14 Dec 2020 22:01:15 GMT
• Title: Odd-One-Out Representation Learning
• Authors: Salman Mohammadi, Anders Kirk Uhrenholt and Bj{\o}rn Sand Jensen
• Abstract summary: We show that a weakly-supervised downstream task based on odd-one-out observations is suitable for model selection. We also show that a bespoke metric-learning VAE model which performs highly on this task also out-performs other standard unsupervised and a weakly-supervised disentanglement model.
• Score: 1.6822770693792826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The effective application of representation learning to real-world problems requires both techniques for learning useful representations, and also robust ways to evaluate properties of representations. Recent work in disentangled representation learning has shown that unsupervised representation learning approaches rely on fully supervised disentanglement metrics, which assume access to labels for ground-truth factors of variation. In many real-world cases ground-truth factors are expensive to collect, or difficult to model, such as for perception. Here we empirically show that a weakly-supervised downstream task based on odd-one-out observations is suitable for model selection by observing high correlation on a difficult downstream abstract visual reasoning task. We also show that a bespoke metric-learning VAE model which performs highly on this task also out-performs other standard unsupervised and a weakly-supervised disentanglement model across several metrics.

Related papers

• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Harnessing Discrete Representations For Continual Reinforcement Learning [8.61539229796467]
  We investigate the advantages of representing observations as vectors of categorical values within the context of reinforcement learning. We find that, when compared to traditional continuous representations, world models learned over discrete representations accurately model more of the world with less capacity.
  arXiv  Detail & Related papers  (2023-12-02T18:55:26Z)
• Leveraging sparse and shared feature activations for disentangled representation learning [112.22699167017471]
  We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation. We validate our approach on six real world distribution shift benchmarks, and different data modalities.
  arXiv  Detail & Related papers  (2023-04-17T01:33:24Z)
• Probing Representation Forgetting in Supervised and Unsupervised Continual Learning [14.462797749666992]
  Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model. We show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning.
  arXiv  Detail & Related papers  (2022-03-24T23:06:08Z)
• Generalizable Information Theoretic Causal Representation [37.54158138447033]
  We propose to learn causal representation from observational data by regularizing the learning procedure with mutual information measures according to our hypothetical causal graph. The optimization involves a counterfactual loss, based on which we deduce a theoretical guarantee that the causality-inspired learning is with reduced sample complexity and better generalization ability.
  arXiv  Detail & Related papers  (2022-02-17T00:38:35Z)
• Desiderata for Representation Learning: A Causal Perspective [104.3711759578494]
  We take a causal perspective on representation learning, formalizing non-spuriousness and efficiency (in supervised representation learning) and disentanglement (in unsupervised representation learning) This yields computable metrics that can be used to assess the degree to which representations satisfy the desiderata of interest and learn non-spurious and disentangled representations from single observational datasets.
  arXiv  Detail & Related papers  (2021-09-08T17:33:54Z)
• Reinforcement Learning with Prototypical Representations [114.35801511501639]
  Proto-RL is a self-supervised framework that ties representation learning with exploration through prototypical representations. These prototypes simultaneously serve as a summarization of the exploratory experience of an agent as well as a basis for representing observations. This enables state-of-the-art downstream policy learning on a set of difficult continuous control tasks.
  arXiv  Detail & Related papers  (2021-02-22T18:56:34Z)
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation [63.042651834453544]
  We show that the unsupervised learning of disentangled representations is impossible without inductive biases on both the models and the data. We observe that while the different methods successfully enforce properties "encouraged" by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision.
  arXiv  Detail & Related papers  (2020-10-27T10:17:15Z)
• Weakly-Supervised Disentanglement Without Compromises [53.55580957483103]
  Intelligent agents should be able to learn useful representations by observing changes in their environment. We model such observations as pairs of non-i.i.d. images sharing at least one of the underlying factors of variation. We show that only knowing how many factors have changed, but not which ones, is sufficient to learn disentangled representations.
  arXiv  Detail & Related papers  (2020-02-07T16:39:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.