The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

• URL: http://arxiv.org/abs/2606.04280v1
• Date: Tue, 02 Jun 2026 23:08:30 GMT
• Title: The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning
• Abstract summary: We develop a measure-theoretic framework formalizing the diversity condition.<n>We show that the standard full-support von Mises-Fisher setting implies the satisfaction of the diversity condition.<n>Our results clarify how sampling mechanisms and inductive bias interact in contrastive representation learning.
• Score: 2.575874871258672
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning has become a leading paradigm for self-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompletely understood. We develop a measure-theoretic framework formalizing the diversity condition, a support requirement on positive-pair sampling that is necessary for isometric latent recovery. We show that the standard full-support von Mises-Fisher setting implies the satisfaction of the diversity condition and as a consequence global contrastive loss minimizers recover latent geometry up to orthogonal transformation, while restricted conditionals can make non-orthogonal maps attain strictly lower asymptotic contrastive loss. We introduce a support-corrected Information Noise Contrastive Estimation (InfoNCE) variant as a theoretical fix: this correction makes orthogonal latent space recovery achievable but does not uniquely select it. Experiments on synthetic benchmarks validate the identifiability predictions, and CIFAR-10 experiments are consistent with the qualitative prediction that architectural inductive bias becomes more important when sampling diversity is limited. Together, our results clarify how sampling mechanisms and encoder inductive bias interact in contrastive representation learning.

Related papers

• Beyond Homoscedasticity: Decoupled Uncertainty Optimization for Deep Imbalanced Regression [7.011984503405774]
  Deep Imbalanced Regression is pervasive in continuous prediction tasks across diverse modalities.<n>We propose DUO, an uncertainty-aware long-tailed regression framework.<n>We show that DUO achieves the best few-shot bMAE and GM on IMDB-WIKI-DIR, AgeDB-DIR, and AAV2-DIR while remaining competitive on few-shot MAE.
  arXiv  Detail & Related papers  (2026-09-04T11:06:57Z)
• Semi-Supervised Conditional Generative Learning through Stochastic Interpolation and Sufficient Representations [12.299177496689621]
  We propose a semi-supervised framework combining conditional intrinsic with low-dimensional latent representations.<n>RepG decomposes generation into two stages: label-dependent latent sampling and high-dimensional reconstruction.
  arXiv  Detail & Related papers  (2026-07-18T09:21:53Z)
• Non-Parametric Rehearsal Learning via Conditional Mean Embeddings [88.89267783967263]
  We propose a non-parametric rehearsal learning approach for textitavoiding undesired future (AUF) problems.<n>Specifically, we use kernel machinery to reformulate the AUF objective into a unified representation that disentangles desirability modeling from action-induced distributional changes.
  arXiv  Detail & Related papers  (2026-05-09T15:30:52Z)
• Machine Learning for Stress Testing: Uncertainty Decomposition in Causal Panel Prediction [2.289837306672451]
  Regulatory stress testing requires projecting credit losses under hypothetical macroeconomic scenarios.<n>We propose a framework for policy-path counterfactual inference in panels.<n>We validate all results through simulation and semi-synthetic experiments with real unemployment data.
  arXiv  Detail & Related papers  (2026-03-08T03:22:38Z)
• Heterogeneous Uncertainty-Guided Composed Image Retrieval with Fine-Grained Probabilistic Learning [49.28548464288051]
  Composed Image Retrieval (CIR) enables image search by combining a reference image with modification text.<n>In intrinsic noise in CIR triplets incurs intrinsic uncertainty and threatens the model's robustness.<n>This paper introduces a Heterogeneous Uncertainty-Guided (HUG) paradigm to overcome these limitations.
  arXiv  Detail & Related papers  (2026-01-16T16:05:49Z)
• Convergence and Generalization of Anti-Regularization for Parametric Models [0.0]
  Anti-regularization introduces a reward term with a reversed sign into the loss function.<n>We formalize spectral safety conditions and trust-region constraints.<n>We design a lightweight safeguard that combines a projection operator with gradient clipping to guarantee stable intervention.
  arXiv  Detail & Related papers  (2025-08-24T15:34:17Z)
• ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data [5.830619388189558]
  Two key tasks in high-dimensional regularized regression are tuning the regularization strength for accurate predictions and estimating the out-of-sample risk.<n>We introduce a new framework, ROTI-GCV, for reliably performing cross-validation under challenging conditions.
  arXiv  Detail & Related papers  (2024-06-17T15:50:00Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Fairness under Covariate Shift: Improving Fairness-Accuracy tradeoff with few Unlabeled Test Samples [21.144077993862652]
  We operate in the unsupervised regime where only a small set of unlabeled test samples along with a labeled training set is available. We experimentally verify that optimizing with our loss formulation outperforms a number of state-of-the-art baselines. We show that our proposed method significantly outperforms them.
  arXiv  Detail & Related papers  (2023-10-11T14:39:51Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.