BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

• URL: http://arxiv.org/abs/2607.26497v3
• Date: Fri, 31 Jul 2026 12:57:39 GMT
• Title: BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms
• Abstract summary: Retrieval-augmented generation (RAG) spans lexical and dense retrieval, graph-based indexing, and agentic search.<n>We present a controlled study that varies corpus size along 28 strictly nested tiers spanning roughly 450-fold.<n>We measure official accuracy, construction and query tokens, and latency.
• Score: 22.217909582731505
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-augmented generation (RAG) spans lexical and dense retrieval, graph-based indexing, and agentic search, but these paradigms are usually evaluated on different benchmarks at one corpus size, leaving their accuracy-cost scaling unclear. To bridge this gap, we present a controlled study that varies corpus size along 28 strictly nested tiers spanning roughly 450-fold, while holding questions and a fixed bedrock of relevant and adversarial documents unchanged. Under one reader model and one judging protocol, we measure official accuracy, construction and query tokens, and latency. The results reveal a scale-dependent crossover rather than an unconditional winner. File-System Agent leads at the smallest shared tiers, but its sequential exploration costs 39 times more query tokens at the bedrock and becomes less effective as the search space grows. Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale. BM25 also anchors the low-cost end of the Pareto frontier without LLM-based construction. Dense retrieval remains efficient but less accurate, whereas graph-based RAG encounters construction walls before deployment scale and its scalable variants remain below BM25 at shared tiers. Overall, corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it.

Related papers

• WARP: Wasserstein-Aligned RAG for Population Opinions [0.9882967349104086]
  WARP calibrates retrieved evidence to the population's opinion distribution.<n>We develop three variants for dense, sparse, and variable candidate pools.<n> WARP's domain-matched variants reduce distributional error by at least 43% with sub-second latency.
  arXiv  Detail & Related papers  (2026-08-24T06:41:23Z)
• Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA [0.0]
  We study baking documents directly into the weights of a 4-bit Gemma-4-e4b model via LoRA.<n>We find that once adapter capacity is adequate, training-data quality is the dominant lever on closed-book accuracy.
  arXiv  Detail & Related papers  (2026-07-23T23:14:59Z)
• Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale [22.13186135187399]
  In this work, we present the first systematic study of in-context retrieval on two scales practical retrievers demand.<n>We first introduce BlockSearch, a 0.6B LM retriever whose architectural and training modifications improve over prior LM baselines.<n>Motivated by this analysis, we introduce length-aware adjustments to the attention softmax and document-level sparse attention.
  arXiv  Detail & Related papers  (2026-07-01T23:38:25Z)
• Towards Retrieving Interaction Spaces for Agentic Search [62.666902610940525]
  Recent direct corpus (DCI) work shows that agents can interact with the raw corpus through shell tools such as 5.4 interaction and file reads.<n>We argue that the role of retrieval for agentic search is not just to select documents that fit in the LLM context window, but to construct an interaction space.
  arXiv  Detail & Related papers  (2026-06-05T03:47:40Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? [74.69723255239663]
  Grounded Personality Reasoning requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding.<n> MM-OCEAN is produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs.<n>The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%.
  arXiv  Detail & Related papers  (2026-05-21T07:42:47Z)
• Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory [1.081571058570587]
  AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism.<n>We introduce DODOCO to test two assumptions about the workload.<n> scaling EP changes the per-expert max/mean token ratio by at most 5% within every architecture's measurable range.
  arXiv  Detail & Related papers  (2026-05-20T10:14:00Z)
• Measuring Maximum Activations in Open Large Language Models [60.3514350516308]
  We measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.<n>We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size.
  arXiv  Detail & Related papers  (2026-05-15T03:31:51Z)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction [127.64173950476702]
  We study direct corpus interaction (DCI), where an agent searches the raw corpus directly with general-purpose terminal tools.<n>This approach requires no offline indexing and adapts naturally to evolving local corpora.<n>Across IR benchmarks and end-to-end agentic search tasks, this simple setup substantially outperforms strong sparse, dense, and reranking baselines.
  arXiv  Detail & Related papers  (2026-05-03T19:13:11Z)
• Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI [0.0]
  We built EnterpriseDocBench to take a swing at parsing fidelity, indexing efficiency, retrieval relevance, and generation groundedness.<n>We ran three pipelines through it -- BM25, dense embedding, and a hybrid -- all with the same GPT-5 generator.<n>One result that genuinely surprised us: factual accuracy on stated claims is 85.5%, but answer completeness averages 0.40.
  arXiv  Detail & Related papers  (2026-04-29T07:48:41Z)
• Semantic Chameleon: Corpus-Dependent Poisoning Attacks and Defenses in RAG Systems [0.0]
  Retrieval-Augmented Generation (RAG) systems extend large language models with external knowledge sources.<n>In particular, adversaries can poison retrieval corpora so that malicious documents are preferentially retrieved at inference time.<n>We study gradient-guided corpus poisoning attacks against modern RAG pipelines and evaluate retrieval-layer defenses.
  arXiv  Detail & Related papers  (2026-03-10T23:15:13Z)
• Beyond the Needle's Illusion: Decoupled Evaluation of Evidence Access and Use under Semantic Interference at 326M-Token Scale [18.13756357502514]
  We introduce EverMemBench-S (EMB-S), an adversarial NIAH-style benchmark built on a 326M-token MemoryBank.<n>While the full MemoryBank spans 326M tokens for retrieval-based (RAG) evaluation, we evaluate native long-context models only at scales that fit within each model's context window.
  arXiv  Detail & Related papers  (2026-01-28T05:44:00Z)
• Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents [50.212640395029744]
  We study how to allocate reasoning budget in deep search pipelines.<n>Using the BrowseComp-Plus benchmark, we analyze tradeoffs between model scale, reasoning effort, reranking depth, and total token cost.
  arXiv  Detail & Related papers  (2026-01-20T18:38:35Z)
• RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension [65.81339691942757]
  RPC-Bench is a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers.<n>We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts.
  arXiv  Detail & Related papers  (2026-01-14T11:37:00Z)
• Relative Scaling Laws for LLMs [91.73497548097775]
  Scaling laws describe how language models improve with additional data, parameters, and compute.<n>We introduce relative scaling laws, which track how performance gaps between test distributions evolve with scale.<n>These results show that although scaling improves overall performance, it is not a universal equalizer.
  arXiv  Detail & Related papers  (2025-10-28T16:55:22Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Fact or Facsimile? Evaluating the Factual Robustness of Modern Retrievers [34.31192184496381]
  Dense retrievers and rerankers are central to retrieval-augmented generation (RAG) pipelines.<n>We evaluate how much factual competence these components inherit or lose from large language models (LLMs) they are based on.<n>For every embedding model, cosine-similarity scores between queries and correct completions are significantly higher than those for incorrect ones.
  arXiv  Detail & Related papers  (2025-08-28T04:13:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.