A Unified Geometric Framework for Weighted Contrastive Learning

• URL: http://arxiv.org/abs/2605.13943v1
• Date: Wed, 13 May 2026 17:48:36 GMT
• Title: A Unified Geometric Framework for Weighted Contrastive Learning
• Abstract summary: We show that weighted InfoNCE objectives can be interpreted as Distance Geometry Problems.<n>This viewpoint yields exact characterizations of the optimal embeddings for several supervised and weakly supervised objectives.<n>Our results show that the choice of weighting scheme determines whether contrastive learning is geometrically realizable, degenerate, or inconsistent.
• Score: 3.746889836344765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning (CL) aims to preserve relational structure between samples by learning representations that reflect a similarity graph. Yet, the geometry of the resulting embeddings remains poorly understood. Here we show that weighted InfoNCE objectives can be interpreted as Distance Geometry Problems, where the weighting scheme specifies the target geometry to be realized by the representation. This viewpoint yields exact characterizations of the optimal embeddings for several supervised and weakly supervised objectives. In supervised classification, both SupCon and Soft SupCon (a dense relaxation of it where pairs from distinct classes have small non-zero similarity) collapse samples within each class to a single prototype. However, while balanced SupCon recovers the classical regular simplex geometry, class imbalance breaks this symmetry: SupCon induces non-uniform inter-class similarities depending on class sizes, whereas Soft SupCon preserves a regular simplex geometry regardless of class imbalance. In continuous-label settings, our framework reveals a different failure mode: y-Aware CL generally cannot attain its entropic optimum unless the labels lie on a hypersphere, exposing a mismatch between Euclidean label weights and spherical latent similarity. By contrast, geometrically consistent choices such as Euclidean-Euclidean weighting or X-CLR admit unique optimal embeddings. Our results show that the choice of weighting scheme determines whether contrastive learning is geometrically realizable, degenerate, or inconsistent, providing a principled framework for designing contrastive objectives.

Related papers

• Exact-Form Regret for Gradient Descent, Mirror Descent and Follow-the-Regularized-Leader [46.31158375592562]
  Recent work shows that first-order methods control richer action-dependent deviations.<n>We ask for a geometric characterization of the deviations with respect to which online gradient descent, mirror descent, and follow-the-regularized-leader (FTRL) achieve no regret.
  arXiv  Detail & Related papers  (2026-09-08T21:32:02Z)
• The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model [73.29820477295043]
  Contrastive Language-Image Pretraining (CLIP) representations form a semantic embedding space governed by cosine similarity.<n>We propose a principled density model for the CLIP latent space based on Mixtures of von Mises-Fisher (MovMF) distributions.
  arXiv  Detail & Related papers  (2026-07-15T10:05:53Z)
• A Geometric View of SRC: Learning Representations for Stable Residual Inference [0.0]
  Reconstruction-based inference assigns a class by comparing class-wise reconstruction residuals.<n>Sparse Representation Classification (SRC) is a canonical instance whose reliability depends on the geometry of the learned representation.<n>We formalize residual-ordering stability through a residual margin and characterize geometric obstructions.
  arXiv  Detail & Related papers  (2026-05-28T09:33:18Z)
• Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets [9.418984531230462]
  We provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced.<n>For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means.
  arXiv  Detail & Related papers  (2026-05-11T22:18:13Z)
• How Label Imbalance Shapes Geometry: A General Spectral Analysis of Multi-Label Neural Collapse [8.294105732989783]
  This work investigates the phenomenon of Neural Collapse (NC) in multi-label classification.<n>It extends its conceptual framework from multi-class learning to general correlated and imbalanced multi-label settings.
  arXiv  Detail & Related papers  (2026-05-03T14:27:12Z)
• Hyperbolic Enhanced Representation Learning for Incomplete Multi-view Clustering [57.38215918201251]
  We propose HERL, a Hyperbolic Enhanced Representation Learning framework for Incomplete Multi-View Clustering.<n> operating within the Poincaré ball, HERL constructs a structure-aware latent space to enhance representation learning.<n>We show that HERL consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-04-18T10:50:46Z)
• Orthogonalized Policy Optimization:Decoupling Sampling Geometry from Optimization Geometry in RLHF [0.0]
  Large language model alignment objectives are often presented as a collection of distinct algorithms, such as PPO, DPO, IPO, and their variants.<n>In this work, we argue that this diversity obscures a simpler underlying structure.<n>We show that this entanglement is not merely a modeling convenience but a source of systematic instability.
  arXiv  Detail & Related papers  (2026-01-18T13:57:44Z)
• Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU [50.9588132578029]
  This paper investigates machine unlearning in hyperbolic contrastive learning.<n>We adapt Alignment to MERU, a model that embeds images and text in hyperbolic space to better capture semantic hierarchies.<n>Our approach introduces hyperbolic-specific components including entailment calibration and norm regularization that leverage the unique properties of hyperbolic space.
  arXiv  Detail & Related papers  (2025-03-19T12:47:37Z)
• Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning [70.0681902472251]
  We propose a novel contrastive learning framework to learn high-quality graph embeddings in hyperbolic space.<n>Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information.<n>We show that in the hyperbolic space one has to address the leaf- and height-level uniformity related to properties of trees.
  arXiv  Detail & Related papers  (2023-10-27T15:31:42Z)
• Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching [26.994954303270575]
  Supervised contrastive loss (SCL) is a competitive and often superior alternative to the cross-entropy loss for classification. While prior studies have demonstrated that both losses yield symmetric training representations under balanced data, this symmetry breaks under class imbalances. This paper presents an intriguing discovery: the introduction of a ReLU activation at the final layer effectively restores the symmetry in SCL-learned representations.
  arXiv  Detail & Related papers  (2023-06-13T17:55:39Z)
• Imbalance Trouble: Revisiting Neural-Collapse Geometry [27.21274327569783]
  We introduce Simplex-Encoded-Labels Interpolation (SELI) as an invariant characterization of the neural collapse phenomenon. We prove for the UFM with cross-entropy loss and vanishing regularization. We present experiments on synthetic and real datasets that confirm convergence to the SELI geometry.
  arXiv  Detail & Related papers  (2022-08-10T18:10:59Z)
• Geometry Contrastive Learning on Heterogeneous Graphs [50.58523799455101]
  This paper proposes a novel self-supervised learning method, termed as Geometry Contrastive Learning (GCL) GCL views a heterogeneous graph from Euclidean and hyperbolic perspective simultaneously, aiming to make a strong merger of the ability of modeling rich semantics and complex structures. Extensive experiments on four benchmarks data sets show that the proposed approach outperforms the strong baselines.
  arXiv  Detail & Related papers  (2022-06-25T03:54:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.