Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative

• URL: http://arxiv.org/abs/2605.07345v1
• Date: Fri, 08 May 2026 06:48:34 GMT
• Title: Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative
• Abstract summary: Mean-pooled cosine similarity is the default metric for comparing neural representations across languages, modalities, and tasks.<n>Under the anisotropy that characterizes modern transformer representations, mean-pooled cosine grows monotonically in sequence length.<n>We argue that length-invariant metrics such as Centered Kernel Alignment should be the default for cross-representation comparisons.
• Score: 1.5718921092089344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mean-pooled cosine similarity is the default metric for comparing neural representations across languages, modalities, and tasks. We establish that this metric is not length-invariant: under the anisotropy that characterizes modern transformer representations, mean-pooled cosine grows monotonically in sequence length, independent of representational content. Empirically, on HumanEvalPack across four code LLMs, the length ratio alone explains $R^2 = 0.52$--$0.75$ of cross-language "Python proximity," while AST depth and shared-token fraction add less than 3% of explained variance beyond length. Substituting Centered Kernel Alignment (CKA) reduces explained variance by 83% and reverses the sign of the length coefficient ($β_{\mathrm{len}}: +0.86 \to -0.37$). The same pattern holds in Mistral-7B on parallel WMT pairs ($R^2 = 0.23$ EN-FR, $R^2 = 0.33$ EN-DE for cosine; $R^2 < 0.01$ for CKA). In CLIP ViT-B/32, mean-pooling reduces the length effect relative to EOS-pooling ($R^2: 0.21 \to {<}0.01$), as predicted by the theory's dependence on anisotropy. We argue that length-invariant metrics such as CKA should be the default for cross-representation comparisons, and that recent claims of cross-lingual representational convergence built on mean-pooled cosine warrant re-examination.

Related papers

• High-Dimensional Nonparametric Change-Point Detection via Low-Rank Degree-Three Density Projection [2.1991623402625606]
  Distributional changes can be invisible to means and covariances yet appear in skewness, asymmetric interactions, or other third-order structure.<n>We develop a nonparametric change-point method that retains every degree-at-most-three coefficient of a density while avoiding direct density estimation.
  arXiv  Detail & Related papers  (2026-08-16T01:21:55Z)
• Is Spurious Correlation Removal Always Learnable? [56.28155520961125]
  Invariant learning can fail even when the structure is statistically identifiable.<n>We show a conditional computational barrier: under a black-box samplable supervised sparse recovery primitive, there exist emphamplable multi-dimensional environments.<n>Synthetic and real datasets illustrate the predicted gaps and transitions and motivate simple diversity diagnostics.
  arXiv  Detail & Related papers  (2026-06-11T05:49:43Z)
• Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m [0.0]
  Independently trained transformers compute the same function in residual-stream bases that differ by a uniform random rotation.<n>We call this phenomenon polymorphism: same function, mutually unintelligible interior coordinates.<n>The phenomenon is invisible to the standard SAE metric.
  arXiv  Detail & Related papers  (2026-05-23T13:37:59Z)
• Representation Without Reward: A JEPA Audit for LLM Fine-Tuning [1.2691047660244335]
  Joint-embedding predictive architectures (JEPAs) propose that a model should learn more useful abstractions when trained to predict latent representations rather than observed outputs.<n>For autoregressive language-model fine-tuning the principle entails a stricter requirement: the induced hidden-state geometry must reach the language-model head emphand improve the decoded task metric.<n>We test that requirement under a fixed Llama-3.2-1B-Instruct LoRA harness on natural-language-to-regex generation.
  arXiv  Detail & Related papers  (2026-05-14T20:27:32Z)
• The E$Δ$-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality [2.28438857884398]
  E$$-MHC-Geo Transformer is a novel architecture that unifies Manifold-Constrained Hyper-Connections (mHC), Deep Delta Learning (DDL), and the Cayley transform.
  arXiv  Detail & Related papers  (2026-05-07T11:37:23Z)
• Scaling Federated Linear Contextual Bandits via Sketching [49.12000877146222]
  This paper proposes Federated Sketch Contextual Linear Bandits (FSCLB)<n>Experiments on both synthetic and real-world datasets show that FSCLB significantly reduces computational and communication costs by over 90 %.
  arXiv  Detail & Related papers  (2026-05-01T08:22:06Z)
• Robust inverse material design with physical guarantees using the Voigt-Reuss Net [0.0]
  We propose a spectrally normalized surrogate for forward and inverse mechanical homogenization with hard physical guarantees.<n>In 3D linear elasticity on an open dataset of biphasic microstructures, a fully connected Voigt-Reuss net trained on $>!7.5times 105$ FFT-based labels with 236 isotropy-in descriptors.<n>Overall, the Voigt-Reuss net unifies accurate, physically admissible forward prediction with large-batch, constraint-consistent inverse design.
  arXiv  Detail & Related papers  (2025-11-14T15:17:37Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• On the $O(\frac{\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1$ Norm [54.28350823319057]
  This paper considers the RMSProp and its momentum extension and establishes the convergence rate of $frac1Tsum_k=1T.<n>Our convergence rate matches the lower bound with respect to all the coefficients except the dimension $d$.<n>Our convergence rate can be considered to be analogous to the $frac1Tsum_k=1T.
  arXiv  Detail & Related papers  (2024-02-01T07:21:32Z)
• Revisiting the Optimality of Word Lengths [92.70590105707639]
  Communicative cost can be operationalized in different ways. Zipf (1935) posited that wordforms are optimized to minimize utterances' communicative costs.
  arXiv  Detail & Related papers  (2023-12-06T20:41:47Z)
• Neural-network quantum state study of the long-range antiferromagnetic Ising chain [0.771303749110121]
  We investigate quantum phase transitions in the transverse field Ising chain with algebraically decaying long-range (LR) antiferromagnetic interactions. We find that the universal ratio of the SR limit does not hold for $alpha_mathrmLR 2$, implying a deviation in the criticality.
  arXiv  Detail & Related papers  (2023-08-18T17:58:36Z)
• A Law of Robustness beyond Isoperimetry [84.33752026418045]
  We prove a Lipschitzness lower bound $Omega(sqrtn/p)$ of robustness of interpolating neural network parameters on arbitrary distributions. We then show the potential benefit of overparametrization for smooth data when $n=mathrmpoly(d)$. We disprove the potential existence of an $O(1)$-Lipschitz robust interpolating function when $n=exp(omega(d))$.
  arXiv  Detail & Related papers  (2022-02-23T16:10:23Z)
• A Simple Convergence Proof of Adam and Adagrad [74.24716715922759]
  We show a proof of convergence between the Adam Adagrad and $O(d(N)/st)$ algorithms. Adam converges with the same convergence $O(d(N)/st)$ when used with the default parameters.
  arXiv  Detail & Related papers  (2020-03-05T01:56:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.