An Investigation of Representation and Allocation Harms in Contrastive Learning

• URL: http://arxiv.org/abs/2310.01583v1
• Date: Mon, 2 Oct 2023 19:25:37 GMT
• Title: An Investigation of Representation and Allocation Harms in Contrastive Learning
• Authors: Subha Maity, Mayank Agarwal, Mikhail Yurochkin, Yuekai Sun
• Abstract summary: We demonstrate that contrastive learning (CL) tends to collapse representations of minority groups with certain majority groups. We refer to this phenomenon as representation harm and demonstrate it on image and text datasets using the corresponding popular CL methods. We provide a theoretical explanation for representation harm using a neural block model that leads to a representational collapse in a contrastive learning setting.
• Score: 55.42336321517228
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The effect of underrepresentation on the performance of minority groups is known to be a serious problem in supervised learning settings; however, it has been underexplored so far in the context of self-supervised learning (SSL). In this paper, we demonstrate that contrastive learning (CL), a popular variant of SSL, tends to collapse representations of minority groups with certain majority groups. We refer to this phenomenon as representation harm and demonstrate it on image and text datasets using the corresponding popular CL methods. Furthermore, our causal mediation analysis of allocation harm on a downstream classification task reveals that representation harm is partly responsible for it, thus emphasizing the importance of studying and mitigating representation harm. Finally, we provide a theoretical explanation for representation harm using a stochastic block model that leads to a representational neural collapse in a contrastive learning setting.

Related papers

• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Time-Series Contrastive Learning against False Negatives and Class Imbalance [17.43801009251228]
  We conduct theoretical analysis and find they have overlooked the fundamental issues: false negatives and class imbalance inherent in the InfoNCE loss-based framework. We introduce a straightforward modification grounded in the SimCLR framework, universally to models engaged in the instance discrimination task. We perform semi-supervised consistency classification and enhance the representative ability of minority classes.
  arXiv  Detail & Related papers  (2023-12-19T08:38:03Z)
• Understanding Self-Supervised Learning of Speech Representation via Invariance and Redundancy Reduction [0.45060992929802207]
  Self-supervised learning (SSL) has emerged as a promising paradigm for learning flexible speech representations from unlabeled data. This study provides an empirical analysis of Barlow Twins (BT), an SSL technique inspired by theories of redundancy reduction in human perception.
  arXiv  Detail & Related papers  (2023-09-07T10:23:59Z)
• On Higher Adversarial Susceptibility of Contrastive Self-Supervised Learning [104.00264962878956]
  Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. It is still largely unknown if the nature of the representation induced by the two learning paradigms is similar. We identify the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon. We devise strategies that are simple, yet effective in improving model robustness with CSL training.
  arXiv  Detail & Related papers  (2022-07-22T03:49:50Z)
• Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
  We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
  arXiv  Detail & Related papers  (2022-05-23T17:38:30Z)
• Fair Contrastive Learning for Facial Attribute Classification [25.436462696033846]
  We propose a new Fair Supervised Contrastive Loss (FSCL) for fair visual representation learning. In this paper, we for the first time analyze unfairness caused by supervised contrastive learning. Our method is robust to the intensity of data bias and effectively works in incomplete supervised settings.
  arXiv  Detail & Related papers  (2022-03-30T11:16:18Z)
• Generalizable Information Theoretic Causal Representation [37.54158138447033]
  We propose to learn causal representation from observational data by regularizing the learning procedure with mutual information measures according to our hypothetical causal graph. The optimization involves a counterfactual loss, based on which we deduce a theoretical guarantee that the causality-inspired learning is with reduced sample complexity and better generalization ability.
  arXiv  Detail & Related papers  (2022-02-17T00:38:35Z)
• Contrastive Learning for Fair Representations [50.95604482330149]
  Trained classification models can unintentionally lead to biased representations and predictions. Existing debiasing methods for classification models, such as adversarial training, are often expensive to train and difficult to optimise. We propose a method for mitigating bias by incorporating contrastive learning, in which instances sharing the same class label are encouraged to have similar representations.
  arXiv  Detail & Related papers  (2021-09-22T10:47:51Z)
• Deep Clustering by Semantic Contrastive Learning [67.28140787010447]
  We introduce a novel variant called Semantic Contrastive Learning (SCL) It explores the characteristics of both conventional contrastive learning and deep clustering. It can amplify the strengths of contrastive learning and deep clustering in a unified approach.
  arXiv  Detail & Related papers  (2021-03-03T20:20:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.