Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms

• URL: http://arxiv.org/abs/2606.30625v1
• Date: Mon, 29 Jun 2026 17:55:40 GMT
• Title: Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms
• Abstract summary: We show that embedding length naturally encodes this information as a byproduct of the training process.<n>We also show how this gives rise to signals that can serve as "free" calibration tools in specific models and retrieval tasks.
• Score: 14.21826317058059
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes. However, surprisingly, empirical studies reveal that despite this, these "discarded" norms seem to correlate with semantic properties such as concept specificity, token frequency, and human uncertainty. In this work, we provide a formal theoretical framework explaining this phenomenon. By analyzing the optimization dynamics, we derive an analytic formula demonstrating that embedding length naturally encodes this information as a byproduct of the training process. We also show how this gives rise to signals that can serve as "free" calibration tools in specific models and retrieval tasks, providing a grounded explanation for a previously heuristic observation.

Related papers

• When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability [1.2314765641075438]
  We introduce a weight-based metric, tensor similarity, that is invariant to such symmetries.<n> Empirically, tensor similarity tracks functional training dynamics, such as grokking and backdoor insertion, with higher fidelity than existing metrics.
  arXiv  Detail & Related papers  (2026-05-14T17:58:27Z)
• Invariant Representations via Wasserstein Correlation Maximization [0.6554326244334866]
  This work investigates the use of Wasserstein correlation for unsupervised representation learning.<n>We find that an (auto)encoder trained to maximize correlation between the input and distributions instead acts as a compressor.<n>We also establish various theoretical results for optimal transport-based dependence measures.
  arXiv  Detail & Related papers  (2025-05-16T21:11:51Z)
• Analyzing Generative Models by Manifold Entropic Metrics [8.477943884416023]
  We introduce a novel set of tractable information-theoretic evaluation metrics.<n>We compare various normalizing flow architectures and $beta$-VAEs on the EMNIST dataset.<n>The most interesting finding of our experiments is a ranking of model architectures and training procedures in terms of their inductive bias to converge to aligned and disentangled representations during training.
  arXiv  Detail & Related papers  (2024-10-25T09:35:00Z)
• Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies [58.179981892921056]
  This work introduces a novel principle for disentanglement we call mechanism sparsity regularization. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors. We show that the latent factors can be recovered by regularizing the learned causal graph to be sparse.
  arXiv  Detail & Related papers  (2024-01-10T02:38:21Z)
• Gradient-Based Feature Learning under Structured Data [57.76552698981579]
  In the anisotropic setting, the commonly used spherical gradient dynamics may fail to recover the true direction. We show that appropriate weight normalization that is reminiscent of batch normalization can alleviate this issue. In particular, under the spiked model with a suitably large spike, the sample complexity of gradient-based training can be made independent of the information exponent.
  arXiv  Detail & Related papers  (2023-09-07T16:55:50Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance [72.50214227616728]
  Interpretability methods are valuable only if their explanations faithfully describe the explained model. We consider neural networks whose predictions are invariant under a specific symmetry group.
  arXiv  Detail & Related papers  (2023-04-13T17:59:03Z)
• Nonparametric Probabilistic Regression with Coarse Learners [1.8275108630751844]
  We show that we can compute precise conditional densities with minimal assumptions on the shape or form of the density. We demonstrate this approach on a variety of datasets and show competitive performance, particularly on larger datasets.
  arXiv  Detail & Related papers  (2022-10-28T16:25:26Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• Instance-Based Neural Dependency Parsing [56.63500180843504]
  We develop neural models that possess an interpretable inference process for dependency parsing. Our models adopt instance-based inference, where dependency edges are extracted and labeled by comparing them to edges in a training set.
  arXiv  Detail & Related papers  (2021-09-28T05:30:52Z)
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective [10.515544361834241]
  State-of-the-art deep learning classifiers are highly susceptible to infinitesmal adversarial perturbations. We find that the learned model is susceptible to adversaries in an intermediate regime where classification generalizes but regression does not. Despite the adversarial susceptibility, we find that classification with these features can be easier than the more commonly studied "independent feature" models.
  arXiv  Detail & Related papers  (2021-09-27T17:35:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.