When Should Active RAG Retrieve? A Budget-Aware Evaluation of Utility, Calibration, and Cost

• URL: http://arxiv.org/abs/2607.24010v1
• Date: Mon, 27 Jul 2026 05:17:06 GMT
• Title: When Should Active RAG Retrieve? A Budget-Aware Evaluation of Utility, Calibration, and Cost
• Abstract summary: Budget-aware Active RAG evaluations should report frontiers, realized usage, threshold-transfer error, harm rates, and cost decompositions alongside accuracy.<n>We operationalize three questions that single-point evaluations conflate: whether trigger scores rank useful retrieval decisions, whether thresholds calibrated on past data meet future budgets, and how trigger-side changes deployment cost.
• Score: 11.216720435946804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Active RAG systems decide when to retrieve external knowledge during generation, making them a budget-sensitive case of agentic RAG and self-adaptive retrieval. Yet evaluations often leave the operating point underspecified: two systems may both claim a 50% evidence-usage budget while realizing different held-out usage rates, so higher accuracy can reflect a looser budget rather than a better retrieval policy. We study budget-aware evaluation for Active RAG by recasting active retrieval as utility estimation, where retrieval is valuable only through its marginal correctness change over a no-retrieval answer. This view separates three questions that single-point evaluations conflate: whether trigger scores rank useful retrieval decisions, whether thresholds calibrated on past data meet future budgets, and how trigger-side computation changes deployment cost. We operationalize these questions with exact top-k utility frontiers, deployable threshold frontiers, conservative budget frontiers, harm audits, and cost decompositions. Across knowledge-intensive multi-hop QA datasets and open instruction models, retrieval harm is non-negligible, router rankings change across datasets and budgets, nominal thresholds can miss target usage, and simple uncertainty or retrieval-score baselines often rival learned utility routers. Budget-aware Active RAG evaluations should therefore report frontiers, realized usage, threshold-transfer error, harm rates, and cost decompositions alongside accuracy.

Related papers

• Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation [64.51088225051959]
  Heterogeneous AI systems can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost.<n>We formalize this tradeoff as an instance of Pandora's Box, the classical problem of optimal search with costly inspection.<n>Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions that determine, for each specialist and input, whether refining the value estimate is worth its cost.
  arXiv  Detail & Related papers  (2026-08-20T17:54:37Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search [0.7243632426715941]
  Agentic Retrieval-Augmented Generation (RAG) systems combine iterative search, planning prompts, and retrievals, but deployed settings impose explicit budgets on tool calls and completion tokens.<n>We present a controlled measurement study of how search depth, retrieval strategy, and completion budget affect accuracy and cost under fixed constraints.
  arXiv  Detail & Related papers  (2026-03-09T19:42:21Z)
• Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generation [55.47971671635531]
  Large Language Models (LLMs) have shown remarkable performance on general Question Answering (QA)<n>Retrieval-Augmented Generation (RAG) addresses this limitation by enriching LLMs with external knowledge.<n>Existing systems primarily rely on unstructured documents, while largely overlooking relational databases.
  arXiv  Detail & Related papers  (2025-09-30T22:19:44Z)
• Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards [110.47944672736723]
  We argue that gains are often overstated due to three forces - an RLVR tax, evaluation pitfalls, and data contamination.<n>We propose a tax-aware training and evaluation protocol that co-optimizes accuracy, grounding, and calibrated abstention.
  arXiv  Detail & Related papers  (2025-09-26T05:06:25Z)
• Cost-aware Stopping for Bayesian Optimization [46.95172329282389]
  We propose a cost-aware stopping rule for Bayesian optimization that adapts to varying evaluation costs and is free of tuning.<n>We prove a theoretical guarantee bounding the expected cumulative evaluation cost incurred by our stopping rule when paired with state-of-the-art acquisition functions.
  arXiv  Detail & Related papers  (2025-07-16T17:54:14Z)
• Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control [52.405085773954596]
  Retrieval-Augmented Generation has emerged as a powerful approach to mitigate large language model hallucinations.<n>Existing RAG frameworks often apply retrieval indiscriminately,leading to inefficiencies-over-retrieving.<n>We introduce a novel user-controllable RAG framework that enables dynamic adjustment of the accuracy-cost trade-off.
  arXiv  Detail & Related papers  (2025-02-17T18:56:20Z)
• Knowledge Retrieval Based on Generative AI [4.9328530417790954]
  This study develops a question-answering system based on Retrieval-Augmented Generation (RAG) using Chinese Wikipedia and Lawbank as retrieval sources.<n>The system employs BGE-M3 for dense vector retrieval to obtain highly relevant search results and BGE-reranker to reorder these results based on query relevance.
  arXiv  Detail & Related papers  (2025-01-08T17:29:46Z)
• Cost-Aware Query Policies in Active Learning for Efficient Autonomous Robotic Exploration [0.0]
  This paper analyzes an AL algorithm for Gaussian Process regression while incorporating action cost. Traditional uncertainty metric with a distance constraint best minimizes root-mean-square error over trajectory distance.
  arXiv  Detail & Related papers  (2024-10-31T18:35:03Z)
• Confidence-Budget Matching for Sequential Budgeted Learning [69.77435313099366]
  We formalize decision-making problems with querying budget. We consider multi-armed bandits, linear bandits, and reinforcement learning problems. We show that CBM based algorithms perform well in the presence of adversity.
  arXiv  Detail & Related papers  (2021-02-05T19:56:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.