Improving BM25 Code Retrieval Under Fixed Generic Tokenization: Adaptive q-Log Odds as a Drop-In BM25 Fix

• URL: http://arxiv.org/abs/2605.18561v1
• Date: Mon, 18 May 2026 15:41:01 GMT
• Title: Improving BM25 Code Retrieval Under Fixed Generic Tokenization: Adaptive q-Log Odds as a Drop-In BM25 Fix
• Abstract summary: Under frozen generic tokenization, a BM25 index has been built by a search system whose analyzer the practitioner does not control.<n>We replace the outer logarithm of the Robertson-Sprck-Jones odds with a q-logarithm.<n>A tokenizer ablation shows that identifier-aware tokenization largely removes the incremental gain from q-IDF.
• Score: 0.2864713389096699
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In retrieval-augmented coding, failures often begin when the relevant file is absent from the retrieved context. Under frozen generic tokenization, where a BM25 index has been built by a search system whose analyzer the practitioner does not control, this failure is routine: BM25's logarithmic RSJ-odds IDF under-separates the identifier tail that distinguishes one function from another. We replace the outer logarithm of the Robertson-Spärck-Jones odds with a q-logarithm. At q=1 the transform recovers BM25 exactly by L'Hôpital's rule, and for q<1 it is a Box-Cox transform of the RSJ odds with lambda = 1-q. On CoIR CodeSearchNet Go (182K documents), oracle-tuned NDCG@10 rises from 0.2575 to 0.4874 (absolute +0.2299; +89.3% relative; zero sign reversals in 10,000 paired-bootstrap resamples, reported as p <= 10^-4). The effect is graded across code languages and is near-zero on BEIR text. A one-parameter closed form estimates a corpus-level q from hapax density and stays near q=1 on corpora where BM25 is already optimal. The index-time cost is a single pass over the sparse score matrix and query latency is unchanged. A tokenizer ablation shows that identifier-aware tokenization largely removes the incremental gain from q-IDF.

Related papers

• CeQe: Grounding Lexical Retrieval in Semantic Evidence [0.914632796153174]
  Cross-Encoder Query Expansion (CE-QE) reads the per-token relevance attributions of a cross-encoder applied to top semantic search results.<n>Every CE-QE expansion term is copied verbatim from a retrieved passage, so it cannot introduce vocabulary the corpus does not contain.<n>On seven BEIR datasets, CE-QE improves lexical recall substantially where query and answer vocabulary diverge.
  arXiv  Detail & Related papers  (2026-08-01T05:27:20Z)
• What Does a Discrete Diffusion Model Learn? [71.03603607324338]
  discrete diffusion model learns: a denoiser, a score ratio, or a bridge plug-in predictor.<n>We start with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process.<n>Every identity is verified numerically, without approximation, on an exactly solvable model.
  arXiv  Detail & Related papers  (2026-07-06T17:56:11Z)
• Transfer Learning for FHIR Questionnaire Terminology Binding [0.0]
  Given a Questionnaire item's text, find the correct LOINC code in a pool of 97,314 active codes.<n>We compare six methods (TF-IDF, frozen MiniLM, BioBERT, BioLORD, contrastively fine-tuned MiniLM, and a TF-IDF+GPT reranker) on a 54-item evaluation set spanning three query styles.
  arXiv  Detail & Related papers  (2026-06-13T19:46:44Z)
• Decoded Quantum Interferometry Beyond Hamming: Rank-Metric and Translation Association Schemes [39.146761527401424]
  Decoded Quantum Interferometry (DQI) uses coherent decoding and a quantum Fourier transform to find high-quality solutions of structured optimisation problems.<n>We extend the core DQI mechanism beyond Hamming space to finite geometries with translation symmetry.<n>We find solutions with an effective-rank proxy near min(m,n)-l, and the corresponding expected score can be converted into a constant-probability bound on the residual rank of a sample.
  arXiv  Detail & Related papers  (2026-06-03T13:12:15Z)
• Entity-Collision: A Stratified Protocol for Attributing Retrieval Lift in Agent Memory [0.0]
  End-to-end agent-memory benchmarks report a single hit@k per retriever.<n>We propose entity-collision, a system-agnostic protocol that pins the BM25 floor by construction.
  arXiv  Detail & Related papers  (2026-05-28T09:02:48Z)
• Entropy Across the Bridge: Conditional-Marginal Discretization for Flow and Schrödinger Samplers [68.9946633641494]
  Flow matching and Schrdinger bridges define probability, yet their inference grids are usually or inherited from one-endpoint diffusion.<n>We derive a conditional-marginal entropy-rate objective for bridge-aware discretization, separating endpoint-conditioned bridge geometry from marginal flow evolution.<n>On EDM/CIFAR-10, the entropic time-discretization gives the best tested five-step FID (186.3 pm 4.0 versus 200.5 pm 2.9 for linear and 238.0 pm 5.3 for cosine)
  arXiv  Detail & Related papers  (2026-05-15T16:11:10Z)
• The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It [0.0]
  Large language models often fail at simple counting tasks, even when the items to count are explicitly present in the prompt.<n>We investigate whether this failure occurs because transformers do not represent counts internally, or because they cannot convert those representations into the correct output tokens.<n>We find strong evidence for the second explanation. Linear probes recover the correct count from intermediate layers with near-perfect accuracy.
  arXiv  Detail & Related papers  (2026-05-05T01:13:06Z)
• Kernel Affine Hull Machines for Compute-Efficient Query-Side Semantic Encoding [1.4021008554367664]
  Transformer-based semantic retrieval is highly effective, yet in many deployments the dominant cost lies in online query encoding rather than corpus indexing.<n>We study the fixed-teacher query-adaptation problem and ask whether repeated neural inference can be replaced by a lightweight, analytically explicit estimator without degrading decision-relevant retrieval quality.<n>We propose Kernel Affine Hull Machines (KAHMs), which map inexpensive lexical features into a frozen semantic embedding space by estimating prototype-mixture weights in a rigorously specified RKHS.<n>On a controlled Austrian-law benchmark (5,000 queries; 84 laws; 10,762 units), KAHM attains the strongest
  arXiv  Detail & Related papers  (2026-05-01T17:46:26Z)
• AXELRAM: Quantize Once, Never Dequantize [0.0]
  AXELRAM computes attention scores directly from quantized KV cache indices without dequantization.<n>We trace the root cause to layer-wise norm and propose a gradient-free sign pattern selection (200 candidates, 8 calibration samples, one-time) that eliminates catastrophic spikes with zero additional hardware cost.
  arXiv  Detail & Related papers  (2026-04-03T02:03:38Z)
• TARG: Training-Free Adaptive Retrieval Gating for Efficient RAG [46.122203287541005]
  Training-free Adaptive Retrieval Gating (TARG) is a single-shot policy that decides when to retrieve using only a short, no-latency draft from the base model.<n>On NQ-Open, TriviaQA, and PopQA, TARG consistently shifts the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2025-11-12T23:09:52Z)
• No Compute Left Behind: Rethinking Reasoning and Sampling with Masked Diffusion Models [42.158430350703505]
  Masked diffusion language models are trained to in-fill positions in randomly masked sequences.<n>We propose reasoning-as-infilling and multi-token entropy decoding.<n>Our work demonstrates that the training and compute used by MDLMs unlock many new inference and post-training methods.
  arXiv  Detail & Related papers  (2025-10-22T19:41:27Z)
• Logistic-Gated Operators Enable Auditable Unit-Aware Thresholds in Symbolic Regression [3.2657452368624678]
  logistic-gated operators (LGO) are differentiable gates with learnable location and steepness.<n>On predominantly smooth tasks, gates are pruned, preserving parsimony.<n>The result is compact symbolic equations with explicit, unit-aware thresholds that can be audited against clinical anchors.
  arXiv  Detail & Related papers  (2025-10-05T16:04:47Z)
• Quantile Advantage Estimation for Entropy-Safe Reasoning [44.192277495613695]
  Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between entropy collapse and entropy explosion<n>We trace both hazards to the mean baseline used in value-free RL, which improperly penalizes negative-advantage samples under reward outliers.<n>We propose Quantile Advantage Estimation (QAE), replacing the mean with a group-wise K-quantile baseline.
  arXiv  Detail & Related papers  (2025-09-26T17:37:52Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Estimating the Decoding Failure Rate of Binary Regular Codes Using Iterative Decoding [84.0257274213152]
  We propose a new technique to provide accurate estimates of the DFR of a two-iterations (parallel) bit flipping decoder.<n>We validate our results, providing comparisons of the modeled and simulated weight of the syndrome, incorrectly-guessed error bit distribution at the end of the first iteration, and two-itcrypteration Decoding Failure Rates (DFR)
  arXiv  Detail & Related papers  (2024-01-30T11:40:24Z)
• Injecting Domain Adaptation with Learning-to-hash for Effective and Efficient Zero-shot Dense Retrieval [49.98615945702959]
  We evaluate LTH and vector compression techniques for improving the downstream zero-shot retrieval accuracy of the TAS-B dense retriever. Our results demonstrate that, unlike prior work, LTH strategies when applied naively can underperform the zero-shot TAS-B dense retriever on average by up to 14% nDCG@10.
  arXiv  Detail & Related papers  (2022-05-23T17:53:44Z)
• Differentially Quantized Gradient Methods [53.3186247068836]
  We show that Differentially Quantized Gradient Descent (DQ-GD) attains a linear contraction factor of $maxsigma_mathrmGD, rhon 2-R$. No algorithm within a certain class can converge faster than $maxsigma_mathrmGD, 2-R$.
  arXiv  Detail & Related papers  (2020-02-06T20:40:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.