Conditional Contrastive Learning: Removing Undesirable Information in Self-Supervised Representations

• URL: http://arxiv.org/abs/2106.02866v1
• Date: Sat, 5 Jun 2021 10:51:26 GMT
• Title: Conditional Contrastive Learning: Removing Undesirable Information in Self-Supervised Representations
• Authors: Yao-Hung Hubert Tsai, Martin Q. Ma, Han Zhao, Kun Zhang, Louis-Philippe Morency, Ruslan Salakhutdinov
• Abstract summary: We develop conditional contrastive learning to remove undesirable information in self-supervised representations. We demonstrate empirically that our methods can successfully learn self-supervised representations for downstream tasks.
• Score: 108.29288034509305
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning is a form of unsupervised learning that leverages rich information in data to learn representations. However, data sometimes contains certain information that may be undesirable for downstream tasks. For instance, gender information may lead to biased decisions on many gender-irrelevant tasks. In this paper, we develop conditional contrastive learning to remove undesirable information in self-supervised representations. To remove the effect of the undesirable variable, our proposed approach conditions on the undesirable variable (i.e., by fixing the variations of it) during the contrastive learning process. In particular, inspired by the contrastive objective InfoNCE, we introduce Conditional InfoNCE (C-InfoNCE), and its computationally efficient variant, Weak-Conditional InfoNCE (WeaC-InfoNCE), for conditional contrastive learning. We demonstrate empirically that our methods can successfully learn self-supervised representations for downstream tasks while removing a great level of information related to the undesirable variables. We study three scenarios, each with a different type of undesirable variables: task-irrelevant meta-information for self-supervised speech representation learning, sensitive attributes for fair representation learning, and domain specification for multi-domain visual representation learning.

Related papers

• Leveraging Superfluous Information in Contrastive Representation Learning [0.0]
  We show that superfluous information does exist during the conventional contrastive learning framework. We design a new objective, namely SuperInfo, to learn robust representations by a linear combination of both predictive and superfluous information. We demonstrate that learning with our loss can often outperform the traditional contrastive learning approaches on image classification, object detection and instance segmentation tasks.
  arXiv  Detail & Related papers  (2024-08-19T16:21:08Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Ignorance is Bliss: Robust Control via Information Gating [60.17644038829572]
  Informational parsimony provides a useful inductive bias for learning representations that achieve better generalization by being robust to noise and spurious correlations. We propose textitinformation gating as a way to learn parsimonious representations that identify the minimal information required for a task.
  arXiv  Detail & Related papers  (2023-03-10T18:31:50Z)
• FUNCK: Information Funnels and Bottlenecks for Invariant Representation Learning [7.804994311050265]
  We investigate a set of related information funnels and bottleneck problems that claim to learn invariant representations from the data. We propose a new element to this family of information-theoretic objectives: The Conditional Privacy Funnel with Side Information. Given the generally intractable objectives, we derive tractable approximations using amortized variational inference parameterized by neural networks.
  arXiv  Detail & Related papers  (2022-11-02T19:37:55Z)
• Rethinking Minimal Sufficient Representation in Contrastive Learning [28.83450836832452]
  We show that contrastive learning models have the risk of over-fitting to the shared information between views. We propose to increase the mutual information between the representation and input as regularization to approximately introduce more task-relevant information. It significantly improves the performance of several classic contrastive learning models in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-14T11:17:48Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• Explainable Recommender Systems via Resolving Learning Representations [57.24565012731325]
  Explanations could help improve user experience and discover system defects. We propose a novel explainable recommendation model through improving the transparency of the representation learning process.
  arXiv  Detail & Related papers  (2020-08-21T05:30:48Z)
• Fairness by Learning Orthogonal Disentangled Representations [50.82638766862974]
  We propose a novel disentanglement approach to invariant representation problem. We enforce the meaningful representation to be agnostic to sensitive information by entropy. The proposed approach is evaluated on five publicly available datasets.
  arXiv  Detail & Related papers  (2020-03-12T11:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.