Understanding Self-supervised Contrastive Learning through Supervised Objectives

• URL: http://arxiv.org/abs/2510.10572v1
• Date: Sun, 12 Oct 2025 12:43:03 GMT
• Title: Understanding Self-supervised Contrastive Learning through Supervised Objectives
• Authors: Byeongchan Lee,
• Abstract summary: We formulate self-supervised representation learning as an approximation to supervised representation learning objectives.<n>Our derivation naturally introduces the concepts of prototype representation bias and a balanced contrastive loss.<n>We empirically validate the effect of balancing positive and negative pair interactions.
• Score: 2.0305676256390934
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised representation learning has achieved impressive empirical success, yet its theoretical understanding remains limited. In this work, we provide a theoretical perspective by formulating self-supervised representation learning as an approximation to supervised representation learning objectives. Based on this formulation, we derive a loss function closely related to popular contrastive losses such as InfoNCE, offering insight into their underlying principles. Our derivation naturally introduces the concepts of prototype representation bias and a balanced contrastive loss, which help explain and improve the behavior of self-supervised learning algorithms. We further show how components of our theoretical framework correspond to established practices in contrastive learning. Finally, we empirically validate the effect of balancing positive and negative pair interactions. All theoretical proofs are provided in the appendix, and our code is included in the supplementary material.

Related papers

• Unsupervised Synthetic Image Attribution: Alignment and Disentanglement [55.853285140682665]
  We propose a simple yet effective unsupervised method called Alignment and Disentanglement.<n>Specifically, we begin by performing basic concept alignment using contrastive self-supervised learning.<n>Next, we enhance the model's attribution ability by promoting representation disentanglement with the Infomax loss.
  arXiv  Detail & Related papers  (2026-01-30T07:31:53Z)
• A Theoretical Framework for Explaining Reinforcement Learning with Shapley Values [0.0]
  Reinforcement learning agents can achieve super-human performance in complex decision-making tasks, but their behaviour is often difficult to understand and explain.<n>We identify three core explanatory targets that together provide a comprehensive view of reinforcement learning agents.<n>We develop a unified theoretical framework for explaining these three elements of reinforcement learning agents through the influence of individual features that the agent observes in its environment.
  arXiv  Detail & Related papers  (2025-05-12T17:48:28Z)
• Understanding Difficult-to-learn Examples in Contrastive Learning: A Theoretical Framework for Spectral Contrastive Learning [20.53618673620584]
  Unsupervised contrastive learning has shown significant performance improvements in recent years, often approaching or even rivaling supervised learning in various tasks.<n>Previous works have shown that difficult-to-learn examples, which are essential in supervised learning, contribute minimally in unsupervised settings.<n>In this paper, we find that the direct removal of difficult-to-learn examples, although reduces the sample size, can boost the downstream classification performance of contrastive learning.
  arXiv  Detail & Related papers  (2025-01-02T16:17:44Z)
• Rethinking Generalizability and Discriminability of Self-Supervised Learning from Evolutionary Game Theory Perspective [43.510860711231544]
  State-of-the-art self-supervised methods tend to enhance either generalizability or discriminability but not both simultaneously.<n>We propose a novel self-supervised learning method that leverages advancements in reinforcement learning to jointly benefit from the general guidance of EGT.
  arXiv  Detail & Related papers  (2024-11-30T17:20:23Z)
• Learning Visual-Semantic Subspace Representations [49.17165360280794]
  We introduce a nuclear norm-based loss function, grounded in the same information theoretic principles that have proved effective in self-supervised learning.<n>We present a theoretical characterization of this loss, demonstrating that, in addition to promoting classity, it encodes the spectral geometry of the data within a subspace lattice.
  arXiv  Detail & Related papers  (2024-05-25T12:51:38Z)
• Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis [10.29814984060018]
  We consider a kernel-based contrastive learning framework termed Kernel Contrastive Learning (KCL) We introduce a formulation of the similarity structure of learned representations by utilizing a statistical dependency viewpoint. We show a new upper bound of the classification error of a downstream task, which explains that our theory is consistent with the empirical success of contrastive learning.
  arXiv  Detail & Related papers  (2023-04-01T21:53:29Z)
• Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
  We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
  arXiv  Detail & Related papers  (2022-05-23T17:38:30Z)
• The Power of Contrast for Feature Learning: A Theoretical Analysis [42.20116348668721]
  We show that contrastive learning outperforms the standard autoencoders and generative adversarial networks. We also illustrate the impact of labeled data in supervised contrastive learning.
  arXiv  Detail & Related papers  (2021-10-06T03:10:28Z)
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control? [80.2534918595143]
  Mutual information provides an appealing formalism for learning representations of data. This paper formalizes the sufficiency of a state representation for learning and representing the optimal policy. Surprisingly, we find that two of these objectives can yield insufficient representations given mild and common assumptions on the structure of the MDP.
  arXiv  Detail & Related papers  (2021-06-14T10:12:34Z)
• Counterfactual Representation Learning with Balancing Weights [74.67296491574318]
  Key to causal inference with observational data is achieving balance in predictive features associated with each treatment type. Recent literature has explored representation learning to achieve this goal. We develop an algorithm for flexible, scalable and accurate estimation of causal effects.
  arXiv  Detail & Related papers  (2020-10-23T19:06:03Z)
• Self-supervised Learning from a Multi-view Perspective [121.63655399591681]
  We show that self-supervised representations can extract task-relevant information and discard task-irrelevant information. Our theoretical framework paves the way to a larger space of self-supervised learning objective design.
  arXiv  Detail & Related papers  (2020-06-10T00:21:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.