Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB

• URL: http://arxiv.org/abs/2511.11041v1
• Date: Fri, 14 Nov 2025 07:51:59 GMT
• Title: Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB
• Abstract summary: We propose a plug-and-play, training-free and lightweight solution called Renormalization.<n>We show that renormalization consistently and statistically significantly improves the performance of existing models.
• Score: 5.315831681082424
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We find that current text embedding models produce outputs with a consistent bias, i.e., each embedding vector $e$ can be decomposed as $\tilde{e} + μ$, where $μ$ is almost identical across all sentences. We propose a plug-and-play, training-free and lightweight solution called Renormalization. Through extensive experiments, we show that renormalization consistently and statistically significantly improves the performance of existing models on the Massive Multilingual Text Embedding Benchmark (MMTEB). In particular, across 38 models, renormalization improves performance by 9.7 $σ$ on retrieval tasks, 3.1 $σ$ on classification tasks, and 0.8 $σ$ on other types of tasks. Renormalization has two variants: directly subtracting $μ$ from $e$, or subtracting the projection of $e$ onto $μ$. We theoretically predict that the latter performs better, and our experiments confirm this prediction.

Related papers

• Reference-Based Bias Detection in LLMs via Relative Representations of Hidden States [4.505185728893222]
  We propose a reference-based method that audits bias in hidden-state representations across related model variants.<n>Because fine-tuning reshapes representation geometry, absolute hidden states are not directly comparable.<n>We measure how target groups shift in their association with positive and negative attributes.
  arXiv  Detail & Related papers  (2026-09-09T11:33:35Z)
• One Good Source is All You Need: Near-Optimal Regret for Bandits under Heterogeneous Noise [49.12618706309658]
  Source-Optimistic Adaptive Regret minimization (SOAR) is a novel algorithm that prunes high-variance sources using sharp variance-concentration bounds.<n>We show it achieves the optimal instance-dependent regret of standard single-source MAB with variance $*2$.<n>Our theoretical bounds represent a significant improvement over some proposed baselines.
  arXiv  Detail & Related papers  (2026-02-16T05:25:06Z)
• CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning [62.56541355300587]
  We introduce a general test-time calibration framework that adaptively modifies the model toward high-reward reasoning paths.<n>Within this framework, we propose CarBoN, a two-phase method that first explores the solution space and then learns a calibration of the logits.<n>Experiments on MATH-500 and AIME-2024 show that CarBoN improves efficiency, with up to $4times$ fewer rollouts to reach the same accuracy.
  arXiv  Detail & Related papers  (2025-10-17T14:04:37Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty [0.3124884279860061]
  elastic net penalty is frequently employed in high-dimensional statistics for parameter regression and variable selection.<n> empirical evidence has shown that the $ell_q$-norm penalty often provides better regression compared to the $ell_r$-norm penalty.<n>We develop two efficient algorithms based on the locally Lipschitz continuous $epsilon$-approximation to $ell_q$-norm.
  arXiv  Detail & Related papers  (2024-11-22T11:55:37Z)
• NeKo: Toward Post Recognition Generative Correction Large Language Models with Task-Oriented Experts [57.53692236201343]
  We propose a Multi-Task Correction MoE, where we train the experts to become an expert'' of speech-to-text, language-to-text and vision-to-text datasets. NeKo performs competitively on grammar and post-OCR correction as a multi-task model.
  arXiv  Detail & Related papers  (2024-11-08T20:11:24Z)
• Transfer Q Star: Principled Decoding for LLM Alignment [105.89114186982972]
  Transfer $Q*$ estimates the optimal value function for a target reward $r$ through a baseline model. Our approach significantly reduces the sub-optimality gap observed in prior SoTA methods.
  arXiv  Detail & Related papers  (2024-05-30T21:36:12Z)
• Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations [44.99833362998488]
  Changes in the data distribution at test time can have deleterious effects on the performance of predictive models. We propose a test-time label shift correction that adapts to changes in the joint distribution $p(y, z)$ using EM applied to unlabeled samples.
  arXiv  Detail & Related papers  (2022-11-28T18:52:33Z)
• The Projected Covariance Measure for assumption-lean variable significance testing [3.8936058127056357]
  A simple but common approach is to specify a linear model, and then test whether the regression coefficient for $X$ is non-zero. We study the problem of testing the model-free null of conditional mean independence, i.e. that the conditional mean of $Y$ given $X$ and $Z$ does not depend on $X$. We propose a simple and general framework that can leverage flexible nonparametric or machine learning methods, such as additive models or random forests.
  arXiv  Detail & Related papers  (2022-11-03T17:55:50Z)
• Ising Model Selection Using $\ell_{1}$-Regularized Linear Regression [13.14903445595385]
  We show that despite model misspecification, the $ell_1$-regularized linear regression ($ell_1$-LinR) estimator can successfully recover the graph structure of the Ising model with $N$ variables. We also provide a computationally efficient method to accurately predict the non-asymptotic performance of the $ell_1$-LinR estimator with moderate $M$ and $N$.
  arXiv  Detail & Related papers  (2021-02-08T03:45:10Z)
• Improving Robustness and Generality of NLP Models Using Disentangled Representations [62.08794500431367]
  Supervised neural networks first map an input $x$ to a single representation $z$, and then map $z$ to the output label $y$. We present methods to improve robustness and generality of NLP models from the standpoint of disentangled representation learning. We show that models trained with the proposed criteria provide better robustness and domain adaptation ability in a wide range of supervised learning tasks.
  arXiv  Detail & Related papers  (2020-09-21T02:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.