Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents

• URL: http://arxiv.org/abs/2608.01913v1
• Date: Mon, 03 Aug 2026 08:46:48 GMT
• Title: Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents
• Abstract summary: Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence.<n>We study these questions through a trajectory-level diagnosis of long-horizon search agents.<n>We evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence.
• Score: 63.2133766267587
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence, but it remains unclear whether and how greater search effort leads to better answers. We study these questions through a trajectory-level diagnosis of long-horizon search agents. Using human-annotated document-level relevance judgments, we evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence. This distinction further allows us to decompose failures into retrieval gaps, where the necessary evidence is never found, and utilization gaps, where relevant evidence is retrieved but not used correctly. With the retrieval model and evaluation harness held fixed, we compare six agents on BrowseComp-Plus and further validate our findings on BrowseComp with an open-web search API. Across settings, we find that search effort and answer quality are only weakly aligned. Answer accuracy is better correlated with the quality of retrieved evidence, especially cumulative retrieval recall, than with the number of searches or the amount of context consumed. Useful evidence often appears early in the trajectory, yet agents tend to continue searching, producing a long tail of low-yield retrieval steps. At the query level, exploratory reformulations remain useful, but the best-performing agents issue far fewer redundant queries. Overall, by systematically characterizing the search behavior and failure modes of long-horizon search agents, this work points to practical directions for building better deep research systems, including stronger query formulation, more effective evidence selection and context management, and stopping criteria based on whether sufficient supporting evidence has been retrieved.

Related papers

• SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task [28.244711705405056]
  Training effective search agents remains challenging due to the lack of scalable and long-horizon tasks.<n>We introduce SearchArt, a framework for training long-horizon search agents.<n>SearchArt constructs large-scale datasets for complex search-, research- and user-oriented tasks.
  arXiv  Detail & Related papers  (2026-07-25T06:47:46Z)
• R$^2$-Searcher: Calibrating Retrieval and Reasoning Boundaries for Agentic Search [59.034878430183305]
  R$2$-Searcher explicitly explores and calibrates the retrieval and reasoning boundaries.<n>We show that R$2$-Searcher significantly outperforms state-of-the-art agentic search methods in answer accuracy and retrieval-reasoning quality.
  arXiv  Detail & Related papers  (2026-06-26T19:44:34Z)
• FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents [79.32220873172078]
  Training deep search agents requires verifiable questions whose answers remain unavailable until sufficient evidence has been acquired through search.<n>We formalize this gap with a shortcut-aware difficulty framework and identify four actionable shortcut risks.<n> Guided by this framework, we introduce FORT, a Framework of Shortcut-Resistant Training-Data Synthesis.
  arXiv  Detail & Related papers  (2026-06-10T13:49:11Z)
• AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning [52.305422887002656]
  We propose a reinforcement learning framework that evaluates each search step via self-generated intermediate answers.<n>By a self-answering mechanism, AutoSearch identifies the minimal sufficient search depth and promotes efficient search.<n>Experiments show that AutoSearch achieves a superior accuracy-efficiency trade-off, alleviating over-searching while preserving search quality.
  arXiv  Detail & Related papers  (2026-04-19T09:05:48Z)
• iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics [9.25340189071758]
  We present iAgentBench, a dynamic ODQA benchmark for cross-source sensemaking.<n>iAgentBench draws seed topics from real-world attention signals and uses common user intent patterns to construct user-like questions.<n>Each instance is released with traceable evidence and auditable intermediate artifacts that support contamination checks.
  arXiv  Detail & Related papers  (2026-03-04T22:40:08Z)
• Over-Searching in Search-Augmented Large Language Models [22.821710825732563]
  Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval.<n>Over-searching leads to computational inefficiency and hallucinations by incorporating irrelevant context.<n>Our finding shows: (i) search generally improves answer accuracy on answerable queries but harms abstention on unanswerable ones; (ii) over-searching is more pronounced in complex reasoning models and deep research systems; and (iii) the composition of retrieved evidence is crucial, as the presence of negative evidence improves abstention.
  arXiv  Detail & Related papers  (2026-01-09T03:24:46Z)
• SmartSearch: Process Reward-Guided Query Refinement for Search Agents [63.46067892354375]
  Large language model (LLM)-based search agents have proven promising for addressing knowledge-intensive problems.<n>Existing works largely focus on optimizing the reasoning paradigms of search agents, yet the quality of intermediate search queries during reasoning remains overlooked.<n>We introduce SmartSearch, a framework built upon two key mechanisms to mitigate this issue.
  arXiv  Detail & Related papers  (2026-01-08T12:39:05Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• Coordinating Search-Informed Reasoning and Reasoning-Guided Search in Claim Verification [16.655011153015202]
  We propose Hierarchical Agent Reasoning and Information Search (HARIS) for multi-hop claim verification.<n>HARIS consists of a high-level reasoning agent that focuses on constructing the main verification chain, generating factual questions when more information is needed, and a low-level search agent that iteratively retrieves more information.<n> Experimental results on the EX-FEVER and HOVER benchmarks demonstrate that HARIS achieves strong performance.
  arXiv  Detail & Related papers  (2025-06-09T08:11:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.