The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-Modal Divergence

• URL: http://arxiv.org/abs/2601.19597v1
• Date: Tue, 27 Jan 2026 13:33:03 GMT
• Title: The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-Modal Divergence
• Abstract summary: We present a measure-theoretic framework that models learning as the evolution of representation measures on a fixed embedding manifold.<n>By establishing value and consistency in the large-batch limit, we bridge the misalignment objective to explicit energy landscapes.<n>We show that this term induces barrier-driven co-adaptation, enforcing a population-level modality gap as a structural geometric necessity.
• Score: 17.501700376593174
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: While InfoNCE powers modern contrastive learning, its geometric mechanisms remain under-characterized beyond the canonical alignment--uniformity decomposition. We present a measure-theoretic framework that models learning as the evolution of representation measures on a fixed embedding manifold. By establishing value and gradient consistency in the large-batch limit, we bridge the stochastic objective to explicit deterministic energy landscapes, uncovering a fundamental geometric bifurcation between the unimodal and multimodal regimes. In the unimodal setting, the intrinsic landscape is strictly convex with a unique Gibbs equilibrium; here, entropy acts merely as a tie-breaker, clarifying "uniformity" as a constrained expansion within the alignment basin. In contrast, the symmetric multimodal objective contains a persistent negative symmetric divergence term that remains even after kernel sharpening. We show that this term induces barrier-driven co-adaptation, enforcing a population-level modality gap as a structural geometric necessity rather than an initialization artifact. Our results shift the analytical lens from pointwise discrimination to population geometry, offering a principled basis for diagnosing and controlling distributional misalignment.

Related papers

• A Theory of Speciation in Generative Diffusion Models on Compact Riemannian Manifolds [0.05729426778193397]
  Speciation in generative diffusion models denotes the emergence of distinct stable branches during denoising.<n>We go beyond existing theoretical descriptions, which usually identify speciation with a symmetric pitchfork bifurcation.<n>A chart-based intrinsic score-learning scheme based on neural networks contrasts the theoretically predicted transitions on prototypal.
  arXiv  Detail & Related papers  (2026-08-24T20:03:31Z)
• A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training [0.0]
  This paper develops a mean-field theory for a single-layer causal multi-head self-attention model trained by cross-entropy minimization.<n>In the infinite-head limit, the averaged attention logits define a risk functional on probability measures, whose first variation generates a nonlinear Wasserstein gradient-flow equation.<n>We study the long-time behavior of the PDE: energy dissipation, convergence to the stationary set under compactness, convergence to a single stationary measure under topological or Kurdyka--ojasiewicz assumptions, and explicit convergence rates under gradient-domination conditions.
  arXiv  Detail & Related papers  (2026-06-09T06:38:27Z)
• Anisotropic Modality Align [91.23979617826926]
  Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data.<n>Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data.<n>The core obstacle lies in the persistent Modality Gap in the shared space.
  arXiv  Detail & Related papers  (2026-05-08T14:53:24Z)
• Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention [0.0]
  Transformer self-attention can be interpreted as a gradient flow on the unit sphere.<n>We develop a theoretical framework for multi-head self-attention dynamics.
  arXiv  Detail & Related papers  (2026-05-05T20:31:11Z)
• The Geometric Price of Discrete Logic: Context-driven Manifold Dynamics of Number Representations [6.414826816896125]
  Large language models (LLMs) generalize smoothly across continuous semantic spaces, yet strict logical reasoning demands the formation of discrete decision boundaries.<n>We argue that task context operates as a non-isometric dynamical operator that enforces a necessary "topological distortion"<n>We validate this geometric evolution across a gradient of tasks, from simple mapping to complex primality testing.
  arXiv  Detail & Related papers  (2026-03-24T13:41:57Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• Manifold Percolation: from generative model to Reinforce learning [0.26905021039717986]
  Generative modeling is typically framed as learning mapping rules, but from an observer's perspective without access to these rules, the task becomes disentangling the geometric support from the probability distribution.<n>We propose that continuum percolation is uniquely suited to this support analysis, as the sampling process effectively projects high-dimensional density estimation onto a geometric counting problem on the support.
  arXiv  Detail & Related papers  (2025-11-25T17:12:42Z)
• Structured Basis Function Networks: Loss-Centric Multi-Hypothesis Ensembles with Controllable Diversity [46.60221265861393]
  Existing approaches to predictive uncertainty rely on multi-hypothesis prediction, which promotes diversity but lacks principled aggregation.<n>The Structured Basis Function Network addresses this gap by linking multi-hypothesis prediction and ensembling through centroidal aggregation induced by Bregman divergences.<n>A tunable diversity mechanism provides parametric control of the bias-variance-diversity trade-off, connecting multi-hypothesis generalisation with loss-aware ensemble aggregation.
  arXiv  Detail & Related papers  (2025-09-02T19:53:43Z)
• Ultracoarse Equilibria and Ordinal-Folding Dynamics in Operator-Algebraic Models of Infinite Multi-Agent Games [0.0]
  We develop an operator algebraic framework for infinite games with a continuum of agents.<n>We prove that regret based learning dynamics governed by a noncommutative continuity equation converge to a unique quantal response equilibrium.<n>We introduce the ordinal folding index, a computable ordinal valued metric that measures the self referential depth of the dynamics.
  arXiv  Detail & Related papers  (2025-07-25T22:20:42Z)
• Spin-only dynamics of the multi-species nonreciprocal Dicke model [0.0]
  Hepp-Lieb-Dicke model is ubiquitous in cavity quantum electrodynamics.<n>We study a variation of the open Dicke model which realizes mediated nonreciprocal interactions between spin species.<n>We find signatures of phase transitions even for small system sizes.
  arXiv  Detail & Related papers  (2025-07-10T17:41:46Z)
• Intrinsic Regularization via Curved Momentum Space: A Geometric Solution to Divergences in Quantum Field Theory [0.0]
  UV divergences in quantum field theory (QFT) have long been a fundamental challenge.<n>We propose a novel and self-consistent approach in which UV regularization emerges naturally from the curved geometry of momentum space.<n>We show seamless extension to Minkowski space, maintainining regularization properties in relativistic QFT.
  arXiv  Detail & Related papers  (2025-02-20T10:49:26Z)
• Self-congruent point in critical matrix product states: An effective field theory for finite-entanglement scaling [6.496379207200742]
  We show that the finite MPS bond dimension $chi$ is equivalent to introducing a perturbation by a relevant operator to the fixed-point Hamiltonian.<n>This phenomenon defines a renormalization group self-congruent point, where the relevant coupling constant ceases to flow due to a balance of two effects.
  arXiv  Detail & Related papers  (2024-11-06T14:35:09Z)
• Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies [58.179981892921056]
  This work introduces a novel principle for disentanglement we call mechanism sparsity regularization. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors. We show that the latent factors can be recovered by regularizing the learned causal graph to be sparse.
  arXiv  Detail & Related papers  (2024-01-10T02:38:21Z)
• Simultaneous Transport Evolution for Minimax Equilibria on Measures [48.82838283786807]
  Min-max optimization problems arise in several key machine learning setups, including adversarial learning and generative modeling. In this work we focus instead in finding mixed equilibria, and consider the associated lifted problem in the space of probability measures. By adding entropic regularization, our main result establishes global convergence towards the global equilibrium.
  arXiv  Detail & Related papers  (2022-02-14T02:23:16Z)
• Geometric phase in a dissipative Jaynes-Cummings model: theoretical explanation for resonance robustness [68.8204255655161]
  We compute the geometric phases acquired in both unitary and dissipative Jaynes-Cummings models. In the dissipative model, the non-unitary effects arise from the outflow of photons through the cavity walls. We show the geometric phase is robust, exhibiting a vanishing correction under a non-unitary evolution.
  arXiv  Detail & Related papers  (2021-10-27T15:27:54Z)
• A Unifying and Canonical Description of Measure-Preserving Diffusions [60.59592461429012]
  A complete recipe of measure-preserving diffusions in Euclidean space was recently derived unifying several MCMC algorithms into a single framework. We develop a geometric theory that improves and generalises this construction to any manifold.
  arXiv  Detail & Related papers  (2021-05-06T17:36:55Z)
• Localisation in quasiperiodic chains: a theory based on convergence of local propagators [68.8204255655161]
  We present a theory of localisation in quasiperiodic chains with nearest-neighbour hoppings, based on the convergence of local propagators. Analysing the convergence of these continued fractions, localisation or its absence can be determined, yielding in turn the critical points and mobility edges. Results are exemplified by analysing the theory for three quasiperiodic models covering a range of behaviour.
  arXiv  Detail & Related papers  (2021-02-18T16:19:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.