Are LLM-Based Retrievers Worth Their Cost? An Empirical Study of Efficiency, Robustness, and Reasoning Overhead

• URL: http://arxiv.org/abs/2604.03676v1
• Date: Sat, 04 Apr 2026 10:24:19 GMT
• Title: Are LLM-Based Retrievers Worth Their Cost? An Empirical Study of Efficiency, Robustness, and Reasoning Overhead
• Abstract summary: We reproduce a reasoning-intensive retrieval benchmark (BRIGHT) across 12 tasks and 14 retrievers.<n>We extend evaluation with cold-start indexing cost, query latency distributions and throughput.<n>We find that some reasoning-specialized retrievers achieve strong effectiveness while remaining competitive in throughput.
• Score: 20.897837226246367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model retrievers improve performance on complex queries, but their practical value depends on efficiency, robustness, and reliable confidence signals in addition to accuracy. We reproduce a reasoning-intensive retrieval benchmark (BRIGHT) across 12 tasks and 14 retrievers, and extend evaluation with cold-start indexing cost, query latency distributions and throughput, corpus scaling, robustness to controlled query perturbations, and confidence use (AUROC) for predicting query success. We also quantify \emph{reasoning overhead} by comparing standard queries to five provided reasoning-augmented variants, measuring accuracy gains relative to added latency. We find that some reasoning-specialized retrievers achieve strong effectiveness while remaining competitive in throughput, whereas several large LLM-based bi-encoders incur substantial latency for modest gains. Reasoning augmentation incurs minimal latency for sub-1B encoders but exhibits diminishing returns for top retrievers and may reduce performance on formal math/code domains. Confidence calibration is consistently weak across model families, indicating that raw retrieval scores are unreliable for downstream routing without additional calibration. We release all code and artifacts for reproducibility.

Related papers

• Test-Time Training for Zero-Resource Dense Retrieval Reranking [3.498051445282382]
  Existing approaches face a fundamental dilemma: cross-encoders deliver strong reranking quality but require costly supervised training and incur high latency.<n>We propose DART (Dense Adaptive Reranking at Test-time), which resolves this dilemma by adapting the scoring function at inference time.<n>On six BEIR benchmarks, DART achieves a mean per-dataset relative NDCG@10 gain of +2.1% over the dense retrieval baseline with under 10ms additional latency per query.
  arXiv  Detail & Related papers  (2026-05-31T07:26:40Z)
• Stabilizing Efficient Reasoning with Step-Level Advantage Selection [54.563811052329235]
  We show that short-context post-training alone, using standard GRPO without any length-aware objective, already induces substantial reasoning compression.<n>We propose Step-level Advantage Selection (SAS), which operates at the reasoning-step level and assigns a zero advantage to low-confidence steps in correct rollouts and to high-confidence steps in verifier-failed rollouts.
  arXiv  Detail & Related papers  (2026-04-27T03:34:59Z)
• Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks [31.017987800426894]
  DREAM is a multi-round debate-based relevance assessment framework with LLM agents.<n>It achieves 95.2% labeling accuracy with only 3.5% human involvement.<n>BRIDGE is a refined benchmark that mitigates evaluation bias and enables fairer retriever comparison.
  arXiv  Detail & Related papers  (2026-02-06T09:27:03Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents [50.212640395029744]
  We study how to allocate reasoning budget in deep search pipelines.<n>Using the BrowseComp-Plus benchmark, we analyze tradeoffs between model scale, reasoning effort, reranking depth, and total token cost.
  arXiv  Detail & Related papers  (2026-01-20T18:38:35Z)
• Your Dense Retriever is Secretly an Expeditious Reasoner [12.123445960145693]
  We propose Adaptive Query Reasoning (AdaQR), a hybrid query rewriting framework.<n>AdaQR reduces reasoning cost by 28% while preserving-or even improving-retrieval performance by 7%.
  arXiv  Detail & Related papers  (2025-09-27T16:50:03Z)
• Self-Aligned Reward: Towards Effective and Efficient Reasoners [16.740993145927774]
  Self-aligned reward (SAR) is a self-guided signal that complements verifiable rewards to encourage both reasoning accuracy and efficiency.<n>We show that SAR reliably distinguishes answer quality: concise, correct answers score higher than redundant ones, and partially correct answers score higher than entirely incorrect ones.
  arXiv  Detail & Related papers  (2025-09-05T20:39:43Z)
• Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination [67.67725938962798]
  Pre-training on massive web-scale corpora leaves Qwen2.5 susceptible to data contamination in widely used benchmarks.<n>We introduce a generator that creates fully clean arithmetic problems of arbitrary length and difficulty, dubbed RandomCalculation.<n>We show that only accurate reward signals yield steady improvements that surpass the base model's performance boundary.
  arXiv  Detail & Related papers  (2025-07-14T17:55:15Z)
• Pseudo Relevance Feedback is Enough to Close the Gap Between Small and Large Dense Retrieval Models [29.934928091542375]
  Scaling dense retrievers to larger large language model (LLM) backbones has been a dominant strategy for improving their retrieval effectiveness.<n>We introduce PromptPRF, a feature-based pseudo-relevance feedback (PRF) framework that enables small LLM-based dense retrievers to achieve effectiveness comparable to much larger models.
  arXiv  Detail & Related papers  (2025-03-19T04:30:20Z)
• Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization [61.02719787737867]
  Large language models (LLMs) are increasingly deployed and democratized on edge devices.<n>One promising solution is uncertainty-based SLM routing, offloading high-stakes queries to stronger LLMs when resulting in low-confidence responses on SLM.<n>We conduct a comprehensive investigation into benchmarking and generalization of uncertainty-driven routing strategies from SLMs to LLMs over 1500+ settings.
  arXiv  Detail & Related papers  (2025-02-06T18:59:11Z)
• An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking [50.81324768683995]
  FIRST is a novel approach that integrates a learning-to-rank objective and leveraging the logits of only the first generated token. We extend the evaluation of FIRST to the TREC Deep Learning datasets (DL19-22), validating its robustness across diverse domains. Our experiments confirm that fast reranking with single-token logits does not compromise out-of-domain reranking quality.
  arXiv  Detail & Related papers  (2024-11-08T12:08:17Z)
• Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers [7.976154147999298]
  inference-free sparse models lag far behind in terms of search relevance when compared to both sparse and dense siamese models.<n>We propose two different approaches for performance improvement.<n>First, we propose an IDF-aware penalty for the matching function that suppresses the contribution of low-IDF tokens.<n>Second, we propose a heterogeneous ensemble knowledge distillation framework that combines siamese dense and sparse retrievers.
  arXiv  Detail & Related papers  (2024-11-07T03:46:43Z)
• FIRST: Faster Improved Listwise Reranking with Single Token Decoding [56.727761901751194]
  First, we introduce FIRST, a novel listwise LLM reranking approach leveraging the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining a robust ranking performance with gains across the BEIR benchmark. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.
  arXiv  Detail & Related papers  (2024-06-21T21:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.