Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings

• URL: http://arxiv.org/abs/2606.29571v1
• Date: Sun, 28 Jun 2026 19:24:59 GMT
• Title: Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings
• Abstract summary: When an encoder spreads its variance evenly across directions, cosine is the best parameter-free choice.<n>When the variance concentrates into a few dominant directions, a property known as anisotropy, rank-based and L1-type metrics beat cosine by a clear margin.<n>One number, the fraction of variance held by the single most dominant dimension, predicts how much the alternatives help across all nineteen encoders.
• Score: 1.14219428942199
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The standard way to compare two text embeddings is cosine similarity. Scattered studies report that a different metric does better, but never pin down the geometric condition that decides when, or why. We settle both with a comprehensive empirical study: nineteen parameter-free similarity metrics on nineteen encoders, from compact sentence transformers up to seven-billion-parameter large language models, across seven datasets. The answer is geometric. When an encoder spreads its variance evenly across directions, cosine is the best parameter-free choice and no other metric helps by a usable margin. When the variance concentrates into a few dominant directions, a property known as anisotropy, rank-based and L1-type metrics beat cosine by a clear margin. The absolute gain is modest, but because cosine starts low on these encoders it is a sizable relative improvement, around twenty percent on average and largest where cosine is weakest. What decides this is the geometry of the embedding space, not how the model was trained: where the two disagree, the metric follows the geometry. One number, the fraction of variance held by the single most dominant dimension, predicts how much the alternatives help across all nineteen encoders, with a rank correlation of 0.86 and a linear correlation of 0.95. To test this as the cause rather than a correlate, we project out the dominant directions: cosine recovers and the advantage of the other metrics nearly vanishes, but only on the encoders that were anisotropic to begin with. The effect is directional, not magnitude based, since it survives normalizing every vector to unit length. Among parameter-free metrics, then, cosine is the right tool wherever an encoder is well spread, which includes the fine-tuned embedders commonly deployed for retrieval, and we give a one-number diagnostic for when it is not.

Related papers

• Beyond Magnitude and Shape: A Direction-Aware Loss for Time Series Forecasting [13.359564942661251]
  We find that MSE-trained forecasters fail on the direction of small moves.<n>We propose CosDir, a simple yet effective direction-aware loss that aligns the difference vectors of the prediction and the target.<n>CosDir is a lightweight, plug-in term that attaches to any backbone without architectural modification.
  arXiv  Detail & Related papers  (2026-08-03T08:06:39Z)
• When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search [61.73232505100331]
  Activation steering offers a lightweight way to control LLMs without retraining, but its effectiveness varies sharply across concepts.<n>We formalize rank-1 steering as a budget-constrained optimization over intervention layer and coefficient.<n>We present textitGRACE, a Granularity- and Representation-Aware Concept Engineering framework that uses activation geometry to diagnose the dominant source of steering difficulty.
  arXiv  Detail & Related papers  (2026-05-09T14:26:49Z)
• Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative [1.5718921092089344]
  Mean-pooled cosine similarity is the default metric for comparing neural representations across languages, modalities, and tasks.<n>Under the anisotropy that characterizes modern transformer representations, mean-pooled cosine grows monotonically in sequence length.<n>We argue that length-invariant metrics such as Centered Kernel Alignment should be the default for cross-representation comparisons.
  arXiv  Detail & Related papers  (2026-05-08T06:48:34Z)
• In Defense of Cosine Similarity: Normalization Eliminates the Gauge Freedom [0.42303492200814446]
  Steck, Ekanadham, and Kallus demonstrate that cosine similarity of learned embeddings can be rendered arbitrary by a diagonal gauge'' matrix $D$.<n>We argue that their conclusion conflates the pathology of an incompatible training objective with the geometric validity of cosine distance on the unit sphere.
  arXiv  Detail & Related papers  (2026-02-23T00:00:57Z)
• Variance-Adjusted Cosine Distance as Similarity Metric [3.776817669946595]
  This study demonstrates limitations of application of cosine similarity.<n>Traditional cosine similarity metric is valid only in the Euclidean space.<n>When there is variance and correlation in the data, then cosine distance is not a completely accurate measure of similarity.
  arXiv  Detail & Related papers  (2025-02-04T11:20:57Z)
• Is Cosine-Similarity of Embeddings Really About Similarity? [46.75365717794515]
  Cosine-similarity is the cosine of the angle between two vectors, or equivalently the dot product between their normalizations. We study embeddings derived from regularized linear models, where closed-form solutions facilitate analytical insights. We derive analytically how cosine-similarity can yield arbitrary and therefore meaningless similarities'
  arXiv  Detail & Related papers  (2024-03-08T16:48:20Z)
• Hyperbolic Vision Transformers: Combining Improvements in Metric Learning [116.13290702262248]
  We propose a new hyperbolic-based model for metric learning. At the core of our method is a vision transformer with output embeddings mapped to hyperbolic space. We evaluate the proposed model with six different formulations on four datasets.
  arXiv  Detail & Related papers  (2022-03-21T09:48:23Z)
• A Triangle Inequality for Cosine Similarity [0.0]
  Similarity search is a fundamental problem for many data analysis techniques. In this paper, we derive a triangle inequality for Cosine similarity that is suitable for efficient similarity search with many standard search structures.
  arXiv  Detail & Related papers  (2021-07-08T19:13:34Z)
• Towards Certified Robustness of Distance Metric Learning [53.96113074344632]
  We advocate imposing an adversarial margin in the input space so as to improve the generalization and robustness of metric learning algorithms. We show that the enlarged margin is beneficial to the generalization ability by using the theoretical technique of algorithmic robustness.
  arXiv  Detail & Related papers  (2020-06-10T16:51:53Z)
• Word Rotator's Distance [50.67809662270474]
  Key principle in assessing textual similarity is measuring the degree of semantic overlap between two texts by considering the word alignment. We show that the norm of word vectors is a good proxy for word importance, and their angle is a good proxy for word similarity. We propose a method that first decouples word vectors into their norm and direction, and then computes alignment-based similarity.
  arXiv  Detail & Related papers  (2020-04-30T17:48:42Z)
• Fast and Robust Comparison of Probability Measures in Heterogeneous Spaces [62.35667646858558]
  We introduce the Anchor Energy (AE) and Anchor Wasserstein (AW) distances, which are respectively the energy and Wasserstein distances instantiated on such representations. Our main contribution is to propose a sweep line algorithm to compute AE emphexactly in log-quadratic time, where a naive implementation would be cubic. We show that AE and AW perform well in various experimental settings at a fraction of the computational cost of popular GW approximations.
  arXiv  Detail & Related papers  (2020-02-05T03:09:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.