When Should LLMs Search? Counterfactual Supervision for Search Routing

• URL: http://arxiv.org/abs/2607.05752v1
• Date: Tue, 07 Jul 2026 02:23:12 GMT
• Title: When Should LLMs Search? Counterfactual Supervision for Search Routing
• Abstract summary: We consider whether search is needed to improve task success relative to a no-search execution.<n>We construct an oracle over NO SEARCH, SEARCH, and UNSOLVED based on task-specific success.<n>We train search-routing policies with supervised fine-tuning and preference optimization.
• Score: 0.04144015562528761
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Search-augmented language models can use external evidence to compensate for limitations in parametric knowledge, but search is not uniformly beneficial: models may call search for questions they can already answer, or rely on noisy evidence when correction, clarification, or abstention would be more appropriate. We formulate this as an instance-level search-routing problem: deciding whether search is needed to improve task success relative to a no-search execution. To derive supervision, we compare no-search and forced-search outcomes for the same question and construct an oracle over NO SEARCH, SEARCH, and UNSOLVED based on task-specific success. Using this oracle as both an evaluation criterion and a learning signal, we train search-routing policies with supervised fine-tuning and preference optimization, improving routing macro-F1 on oracle-eligible examples from 0.7082 to 0.8235 for Gemma E2B and from 0.7053 to 0.8365 for Qwen3.5-4B. Further analysis shows that the learned policies reduce model-specific routing failures: Gemma primarily learns no-search restraint, while Qwen further reduces missed search; residual UNSOLVED cases reveal heterogeneous bottlenecks involving model capacity, retrieval budget, evidence use, and policy behavior.

Related papers

• SearchMaster: Grounded and Regulated Self-Play for Search Agents [57.537666863340284]
  Training search agents requires high-quality search data.<n> Existing pipelines often depend on human-written tasks, expert demonstrations, or stronger teacher models.<n>We present SearchMaster, a self-play framework that trains a single LLM from search tasks it generates, solves, and verifies in a local search environment.
  arXiv  Detail & Related papers  (2026-08-03T07:29:11Z)
• SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach [35.31208793946578]
  Specialised retrieval agents typically surface higher quality results than general-purpose search.<n>Current approaches route queries based on inferred topic or intent, but it does not incorporate signal from retrieved content.<n>We address this by training a small language model via supervised fine-tuning followed by reinforcement learning.
  arXiv  Detail & Related papers  (2026-07-15T11:11:39Z)
• SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Banks [32.240602087886494]
  We propose a framework that makes query planning explicit through reusable search skills.<n>Across open-source and closed-source models, SearchSkill improves exact match on knowledge-intensive QA benchmarks.<n>These results suggest that explicit skill-conditioned query planning is a lightweight alternative to treating search as an undifferentiated action.
  arXiv  Detail & Related papers  (2026-05-09T16:23:54Z)
• SE-Search: Self-Evolving Search Agent via Memory and Dense Reward [87.79131676521656]
  Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge.<n>Existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals.<n>We propose textbfSelf-textbfEvolving textbfSearch, a Self-Evolving Search agent that improves online search behavior through three components.
  arXiv  Detail & Related papers  (2026-02-06T09:14:07Z)
• AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning [61.974530499621274]
  Overreliance on search introduces unnecessary cost and risks exposure to noisy or malicious content.<n>We propose a two-stage, outcome-driven RL framework that disentangles problem solving from the decision of whether to invoke search.<n>AdaSearch substantially improves knowledge-boundary awareness, reduces unnecessary search calls, preserves strong task performance, and offers more transparent, interpretable decision behaviors.
  arXiv  Detail & Related papers  (2025-12-18T18:50:01Z)
• AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play [45.02121903138421]
  AceSearcher trains a single large language model (LLM) to alternate between two roles: a decomposer that breaks down complex queries and a solver that integrates retrieved contexts for answer generation.<n>Experiments on three reasoning-intensive tasks across 10 datasets show that AceSearcher outperforms state-of-the-art baselines, achieving an average exact match improvement of 7.6%.
  arXiv  Detail & Related papers  (2025-09-29T02:14:30Z)
• MMSearch-R1: Incentivizing LMMs to Search [49.889749277236376]
  We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables on-demand, multi-turn search in real-world Internet environments.<n>Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty.
  arXiv  Detail & Related papers  (2025-06-25T17:59:42Z)
• LLM-First Search: Self-Guided Exploration of the Solution Space [29.780554400938335]
  Large Language Models (LLMs) have demonstrated remarkable improvements in reasoning and planning through increased test-time compute.<n>We propose textbfLLM-First Search (LFS), a novel textitLLM Self-Guided Search method.
  arXiv  Detail & Related papers  (2025-06-05T16:27:49Z)
• SEM: Reinforcement Learning for Search-Efficient Large Language Models [26.075903427834838]
  Large Language Models (LLMs) have demonstrated their capabilities not only in reasoning but also in invoking external tools.<n>Existing reinforcement learning approaches often lead to redundant search behaviors, resulting in inefficiencies and over-cost.<n>We propose SEM, a novel post-training reinforcement learning framework that explicitly trains LLMs to optimize search usage.
  arXiv  Detail & Related papers  (2025-05-12T09:45:40Z)
• ZeroSearch: Incentivize the Search Capability of LLMs without Searching [69.55482019211597]
  We introduce ZeroSearch, a framework that incentivizes the capabilities of large language models to use a real search engine with simulated searches during training.<n>Our approach begins with lightweight supervised fine-tuning to transform the LLM into a retrieval module capable of generating both useful and noisy documents.
  arXiv  Detail & Related papers  (2025-05-07T17:30:22Z)
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning [74.65632662894086]
  We propose ReSearch, a framework that trains LLMs to Reason with Search via reinforcement learning.<n>Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking.<n>Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction.
  arXiv  Detail & Related papers  (2025-03-25T09:00:58Z)
• RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering [42.66223628527439]
  Adaptive retrieval-augmented generation (ARAG) aims to dynamically determine the necessity of retrieval for queries instead of retrieving indiscriminately. This work presents a benchmark, RetrievalQA, comprising 1,271 short-form questions covering new world and long-tail knowledge.
  arXiv  Detail & Related papers  (2024-02-26T09:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.