Spectral Tempering for Embedding Compression in Dense Passage Retrieval

• URL: http://arxiv.org/abs/2603.19339v1
• Date: Thu, 19 Mar 2026 10:01:32 GMT
• Title: Spectral Tempering for Embedding Compression in Dense Passage Retrieval
• Abstract summary: We show that the optimal scaling strength $$ is not a global constant.<n>We propose Spectral Tempering, a learning-free method that derives an adaptive $(k)$ directly from the corpus eigenspectrum.
• Score: 17.660889990235656
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dimensionality reduction is critical for deploying dense retrieval systems at scale, yet mainstream post-hoc methods face a fundamental trade-off: principal component analysis (PCA) preserves dominant variance but underutilizes representational capacity, while whitening enforces isotropy at the cost of amplifying noise in the heavy-tailed eigenspectrum of retrieval embeddings. Intermediate spectral scaling methods unify these extremes by reweighting dimensions with a power coefficient $γ$, but treat $γ$ as a fixed hyperparameter that requires task-specific tuning. We show that the optimal scaling strength $γ$ is not a global constant: it varies systematically with target dimensionality $k$ and is governed by the signal-to-noise ratio (SNR) of the retained subspace. Based on this insight, we propose Spectral Tempering (\textbf{SpecTemp}), a learning-free method that derives an adaptive $γ(k)$ directly from the corpus eigenspectrum using local SNR analysis and knee-point normalization, requiring no labeled data or validation-based search. Extensive experiments demonstrate that Spectral Tempering consistently achieves near-oracle performance relative to grid-searched $γ^*(k)$ while remaining fully learning-free and model-agnostic. Our code is publicly available at https://anonymous.4open.science/r/SpecTemp-0D37.

Related papers

• High-Probability Convergence in Decentralized Stochastic Optimization with Gradient Tracking [69.90407799170687]
  We study high-probability (HP) convergence guarantees in decentralized optimization.<n>Our results establish the same conditions on the horizon as in achieving comparable times.
  arXiv  Detail & Related papers  (2026-04-30T22:45:21Z)
• Diffusion-Guided Feature Selection via Nishimori Temperature: Noise-Based Spectral Embedding [0.0]
  We propose Noise-Based Spectral Embedding (NBSE), a framework for selecting informative features from high-dimensional data without greedy search.<n>NBSE constructs a sparse similarity graph on the samples and identifies the Nishimori temperature $_N$ the critical inverse temperature at which the Bethe Hessian becomes singular.<n>By transposing the data matrix and applying NBSE in feature space, we obtain a one-dimensional spectral embedding that reveals groups of redundant or semantically related dimensions.
  arXiv  Detail & Related papers  (2026-04-27T16:53:40Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Phase-space entropy at acquisition reflects downstream learnability [54.4100065023873]
  We propose an acquisition-level scalar $S_mathcal B$ based on instrument-resolved phase space.<n>We show theoretically that (S_mathcal B) correctly identifies the phase-space coherence of periodic sampling.<n>$|S_mathcal B|$ consistently ranks sampling geometries and predicts downstream reconstruction/recognition difficulty emphwithout training.
  arXiv  Detail & Related papers  (2025-12-22T10:03:51Z)
• Spectral Perturbation Bounds for Low-Rank Approximation with Applications to Privacy [13.264499801590583]
  We introduce new high-probability spectral-norm perturbation bounds for symmetric matrix $A in mathbbRn times n$ and an arbitrary symmetric perturbationE$.<n>Under mild eigengap and norm conditions, our bounds yield sharp estimates for $|(A + E)_p - A_p|$, with improvements of up to a factor of $sqrtn$.<n>As an application, we derive improved utility guarantees for differentially private PCA, resolving an open problem in the literature.
  arXiv  Detail & Related papers  (2025-10-29T16:36:00Z)
• deep-REMAP: Probabilistic Parameterization of Stellar Spectra Using Regularized Multi-Task Learning [0.0]
  deep-REMAP is a novel deep learning framework that utilizes a regularized, multi-task approach to predict stellar atmospheric parameters from observed spectra.<n>We train a deep convolutional neural network on the PHOENIX synthetic spectral library and use transfer learning to fine-tune the model on a small subset of observed FGK dwarf spectra.<n>Deep-REMAP accurately recovers the effective temperature ($T_rmeff$), surface gravity ($log rmg$), and metallicity ([Fe/H]), achieving a precision of, for instance, approximately 75 K in $T
  arXiv  Detail & Related papers  (2025-10-10T13:20:06Z)
• Theoretical Framework for Tempered Fractional Gradient Descent: Application to Breast Cancer Classification [0.0]
  This paper introduces Fractional Gradient Descent (TFGD), a novel optimization framework that synergizes fractional calculus with exponential tempering to enhance gradient-based learning.<n>TFGD addresses limitations by incorporating a tempered memory mechanism, where historical gradients are weighted by fractional coefficients $|w_j| = binomalphaj$ and exponentially decayed via a tempering parameter $lambda$.<n> Empirical validation on the Breast Cancer Wisconsin dataset demonstrates TFGD's superiority, achieving 98.25% test accuracy (vs. 92.11% for SGD) and 2$times$ faster convergence.
  arXiv  Detail & Related papers  (2025-04-26T08:26:34Z)
• Truncated Kernel Stochastic Gradient Descent on Spheres [1.4583059436979549]
  Inspired by the structure of spherical harmonics, we propose the truncated kernel gradient descent (T- Kernel SGD) algorithm.<n>T- Kernel SGD has a least-square loss function for spherical data fitting.
  arXiv  Detail & Related papers  (2024-10-02T14:09:51Z)
• Hyperspectral Image Denoising Using Non-convex Local Low-rank and Sparse Separation with Spatial-Spectral Total Variation Regularization [49.55649406434796]
  We propose a novel non particular approach to robust principal component analysis for HSI denoising. We develop accurate approximations to both rank and sparse components. Experiments on both simulated and real HSIs demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-01-08T11:48:46Z)
• Towards Sample-Optimal Compressive Phase Retrieval with Sparse and Generative Priors [59.33977545294148]
  We show that $O(k log L)$ samples suffice to guarantee that the signal is close to any vector that minimizes an amplitude-based empirical loss function. We adapt this result to sparse phase retrieval, and show that $O(s log n)$ samples are sufficient for a similar guarantee when the underlying signal is $s$-sparse and $n$-dimensional.
  arXiv  Detail & Related papers  (2021-06-29T12:49:54Z)
• Spectral density estimation with the Gaussian Integral Transform [91.3755431537592]
  spectral density operator $hatrho(omega)=delta(omega-hatH)$ plays a central role in linear response theory. We describe a near optimal quantum algorithm providing an approximation to the spectral density.
  arXiv  Detail & Related papers  (2020-04-10T03:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.