Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

• URL: http://arxiv.org/abs/2605.11291v1
• Date: Mon, 11 May 2026 22:18:13 GMT
• Title: Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets
• Abstract summary: We provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced.<n>For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means.
• Score: 9.418984531230462
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: In this paper, we provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced. When classes are balanced and the representation dimension is greater than the number of classes, it is well-known that the optimal representations exhibit Neural Collapse (NC), i.e., representations from the same class collapse to their class means and the class means form an Equiangular Tight Frame (ETF). For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means and their geometry exhibits an angular symmetry structure that is determined by the relative class proportions. In general, we show that the geometry can be determined by solving a convex optimization problem. Exploiting this symmetry structure, we analytically investigate a special case where class imbalance is extreme and prove that CL exhibits a phenomenon called Minority Collapse (MC) where all samples from the minority classes (classes with small probabilities) collapse into a single vector, whenever the class imbalance exceeds a threshold, which in turn depends on the regularity properties of the CL loss used and on the number of negative samples. Numerical results are provided to illustrate these phenomena and corroborate the theoretical results. We conclude by identifying a number of open problems.

Related papers

• A Unified Geometric Framework for Weighted Contrastive Learning [3.746889836344765]
  We show that weighted InfoNCE objectives can be interpreted as Distance Geometry Problems.<n>This viewpoint yields exact characterizations of the optimal embeddings for several supervised and weakly supervised objectives.<n>Our results show that the choice of weighting scheme determines whether contrastive learning is geometrically realizable, degenerate, or inconsistent.
  arXiv  Detail & Related papers  (2026-05-13T17:48:36Z)
• Equilibrium contrastive learning for imbalanced image classification [5.198140836737882]
  Contrastive learning (CL) is a predominant technique in image classification, but they showed limited performance with an imbalanced dataset.<n>Recently, several supervised CL methods have been proposed to promote an ideal regular simplex geometric configuration.<n>We propose Equilibrium Contrastive Learning (ECL), a supervised CL framework designed to promote geometric equilibrium.
  arXiv  Detail & Related papers  (2026-02-10T08:07:56Z)
• SIGMA: Scalable Spectral Insights for LLM Collapse [51.863164847253366]
  We introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework for model collapse.<n>By utilizing benchmarks that deriving and deterministic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space.<n>We demonstrate that SIGMA effectively captures the transition towards states, offering both theoretical insights into the mechanics of collapse.
  arXiv  Detail & Related papers  (2026-01-06T19:47:11Z)
• All-around Neural Collapse for Imbalanced Classification [27.467732819969935]
  textbfAll-around textbfNeural textbfCollapse framework (AllNC) aims to comprehensively restore Neural Collapse across multiple aspects including individual activations, class means and classifier vectors. We thoroughly analyze its effectiveness and verify on multiple benchmark datasets that it achieves state-of-the-art in both balanced and imbalanced settings.
  arXiv  Detail & Related papers  (2024-08-14T02:06:24Z)
• Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse [16.42457033976047]
  We prove that the losses of Supervised Contrastive Learning (SCL), Hard-SCL (HSCL), and Unsupervised Contrastive Learning (UCL) are minimized by representations that exhibit Neural-Collapse (NC)<n>We also prove that for any representation mapping, the HSCL and Hard-UCL (HUCL) losses are lower bounded by the corresponding SCL and UCL losses.
  arXiv  Detail & Related papers  (2023-11-09T04:40:32Z)
• Uncertainty-guided Boundary Learning for Imbalanced Social Event Detection [64.4350027428928]
  We propose a novel uncertainty-guided class imbalance learning framework for imbalanced social event detection tasks. Our model significantly improves social event representation and classification tasks in almost all classes, especially those uncertain ones.
  arXiv  Detail & Related papers  (2023-10-30T03:32:04Z)
• Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching [26.994954303270575]
  Supervised contrastive loss (SCL) is a competitive and often superior alternative to the cross-entropy loss for classification. While prior studies have demonstrated that both losses yield symmetric training representations under balanced data, this symmetry breaks under class imbalances. This paper presents an intriguing discovery: the introduction of a ReLU activation at the final layer effectively restores the symmetry in SCL-learned representations.
  arXiv  Detail & Related papers  (2023-06-13T17:55:39Z)
• Predicting and Enhancing the Fairness of DNNs with the Curvature of Perceptual Manifolds [44.79535333220044]
  Recent studies have shown that tail classes are not always hard to learn, and model bias has been observed on sample-balanced datasets.<n>In this work, we first establish a geometric perspective for analyzing model fairness and then systematically propose a series of geometric measurements.
  arXiv  Detail & Related papers  (2023-03-22T04:49:23Z)
• Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning [120.53458753007851]
  Few-shot class-incremental learning (FSCIL) has been a challenging problem as only a few training samples are accessible for each novel class in the new sessions. We deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse. We propose a neural collapse inspired framework for FSCIL. Experiments on the miniImageNet, CUB-200, and CIFAR-100 datasets demonstrate that our proposed framework outperforms the state-of-the-art performances.
  arXiv  Detail & Related papers  (2023-02-06T18:39:40Z)
• Learning Graphical Factor Models with Riemannian Optimization [70.13748170371889]
  This paper proposes a flexible algorithmic framework for graph learning under low-rank structural constraints. The problem is expressed as penalized maximum likelihood estimation of an elliptical distribution. We leverage geometries of positive definite matrices and positive semi-definite matrices of fixed rank that are well suited to elliptical models.
  arXiv  Detail & Related papers  (2022-10-21T13:19:45Z)
• Imbalance Trouble: Revisiting Neural-Collapse Geometry [27.21274327569783]
  We introduce Simplex-Encoded-Labels Interpolation (SELI) as an invariant characterization of the neural collapse phenomenon. We prove for the UFM with cross-entropy loss and vanishing regularization. We present experiments on synthetic and real datasets that confirm convergence to the SELI geometry.
  arXiv  Detail & Related papers  (2022-08-10T18:10:59Z)
• Local and Global Uniform Convexity Conditions [88.3673525964507]
  We review various characterizations of uniform convexity and smoothness on norm balls in finite-dimensional spaces. We establish local versions of these conditions to provide sharper insights on a recent body of complexity results in learning theory, online learning, or offline optimization. We conclude with some practical examples in optimization and machine learning where leveraging these conditions and localized assumptions lead to new complexity results.
  arXiv  Detail & Related papers  (2021-02-09T21:09:53Z)
• Rethinking preventing class-collapsing in metric learning with margin-based losses [81.22825616879936]
  Metric learning seeks embeddings where visually similar instances are close and dissimilar instances are apart. margin-based losses tend to project all samples of a class onto a single point in the embedding space. We propose a simple modification to the embedding losses such that each sample selects its nearest same-class counterpart in a batch.
  arXiv  Detail & Related papers  (2020-06-09T09:59:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.