Information-Preserving Contrastive Learning for Self-Supervised Representations

• URL: http://arxiv.org/abs/2012.09962v2
• Date: Wed, 17 Feb 2021 16:53:53 GMT
• Title: Information-Preserving Contrastive Learning for Self-Supervised Representations
• Authors: Tianhong Li, Lijie Fan, Yuan Yuan, Hao He, Yonglong Tian, Dina Katabi
• Abstract summary: Contrastive learning is very effective at learning useful representations without supervision. It may learn a shortcut that is irrelevant to the downstream task, and discard relevant information. This paper presents information-preserving contrastive learning (IPCL), a new framework for unsupervised representation learning.
• Score: 42.74927142208983
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning is very effective at learning useful representations without supervision. Yet contrastive learning has its limitations. It may learn a shortcut that is irrelevant to the downstream task, and discard relevant information. Past work has addressed this limitation via custom data augmentations that eliminate the shortcut. This solution however does not work for data modalities that are not interpretable by humans, e.g., radio signals. For such modalities, it is hard for a human to guess which shortcuts may exist in the signal, or how they can be eliminated. Even for interpretable data, sometimes eliminating the shortcut may be undesirable. The shortcut may be irrelevant to one downstream task but important to another. In this case, it is desirable to learn a representation that captures both the shortcut information and the information relevant to the other downstream task. This paper presents information-preserving contrastive learning (IPCL), a new framework for unsupervised representation learning that preserves relevant information even in the presence of shortcuts. We empirically show that the representations learned by IPCL outperforms contrastive learning in supporting different modalities and multiple diverse downstream tasks.

Related papers

• Exposing Image Classifier Shortcuts with Counterfactual Frequency (CoF) Tables [0.0]
  'Shortcuts' are easy-to-learn patterns from the training data that fail to generalise to new data. Examples include the use of a copyright watermark to recognise horses, snowy background to recognise huskies, or ink markings to detect malignant skin lesions. We introduce Counterfactual Frequency tables, a novel approach that aggregates instance-based explanations into global insights.
  arXiv  Detail & Related papers  (2024-05-24T15:58:02Z)
• Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning [57.4893889206655]
  We introduce synthetic shortcuts for vision-language: a training and evaluation framework. We show that contrastive VLMs trained from scratch or fine-tuned with data containing these synthetic shortcuts mainly learn features that represent the shortcut.
  arXiv  Detail & Related papers  (2024-02-27T13:50:34Z)
• TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning [73.53576440536682]
  We introduce TACO: Temporal Action-driven Contrastive Learning, a powerful temporal contrastive learning approach. TACO simultaneously learns a state and an action representation by optimizing the mutual information between representations of current states. For online RL, TACO achieves 40% performance boost after one million environment interaction steps.
  arXiv  Detail & Related papers  (2023-06-22T22:21:53Z)
• Ignorance is Bliss: Robust Control via Information Gating [60.17644038829572]
  Informational parsimony provides a useful inductive bias for learning representations that achieve better generalization by being robust to noise and spurious correlations. We propose textitinformation gating as a way to learn parsimonious representations that identify the minimal information required for a task.
  arXiv  Detail & Related papers  (2023-03-10T18:31:50Z)
• Which Shortcut Solution Do Question Answering Models Prefer to Learn? [38.36299280464046]
  Question answering (QA) models for reading comprehension tend to learn shortcut solutions rather than the solutions intended by QA datasets. We show that shortcuts that exploit answer positions and word-label correlations are preferentially learned for extractive and multiple-choice QA. We experimentally show that the learnability of shortcuts can be utilized to construct an effective QA training set.
  arXiv  Detail & Related papers  (2022-11-29T13:57:59Z)
• Rethinking Minimal Sufficient Representation in Contrastive Learning [28.83450836832452]
  We show that contrastive learning models have the risk of over-fitting to the shared information between views. We propose to increase the mutual information between the representation and input as regularization to approximately introduce more task-relevant information. It significantly improves the performance of several classic contrastive learning models in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-14T11:17:48Z)
• Predicting What You Already Know Helps: Provable Self-Supervised Learning [60.27658820909876]
  Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data. We show a mechanism exploiting the statistical connections between certain em reconstruction-based pretext tasks that guarantee to learn a good representation. We prove the linear layer yields small approximation error even for complex ground truth function class.
  arXiv  Detail & Related papers  (2020-08-03T17:56:13Z)
• Evolving Losses for Unsupervised Video Representation Learning [91.2683362199263]
  We present a new method to learn video representations from large-scale unlabeled video data. The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods.
  arXiv  Detail & Related papers  (2020-02-26T16:56:07Z)
• Self-supervised visual feature learning with curriculum [0.24366811507669126]
  This paper takes inspiration from curriculum learning to progressively remove low level signals. It shows that it significantly increase the speed of convergence of the downstream task.
  arXiv  Detail & Related papers  (2020-01-16T03:28:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.