SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

• URL: http://arxiv.org/abs/2608.05212v1
• Date: Wed, 05 Aug 2026 09:09:00 GMT
• Title: SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
• Abstract summary: We introduce SearchAuditBench, a benchmark that evaluates whether auditors can localize, attribute, and repair search-agent failures.<n>SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks.<n>We propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures.
• Score: 56.400332595452966
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. We further propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors.

Related papers

• How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks [6.001213285107465]
  AutoResearchEval features 100 tasks grounded in published frontier science across 7 scientific domains.<n> evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories.<n>We leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts.
  arXiv  Detail & Related papers  (2026-08-14T21:39:38Z)
• ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment [66.45065414461274]
  Long-horizon search agents must make multiple sequential actions (steps) to search, retrieve, verify, and integrate evidence to reach a final answer.<n>We propose Answer-Backtracked Credit Assignment (ABC), a fine-grained credit assignment framework for training long-horizon search agents.<n>ABC converts sparse trajectory-level outcomes into dense step-level supervision that rewards useful actions (even in failed trajectories) while suppressing erroneous or redundant actions.
  arXiv  Detail & Related papers  (2026-08-05T17:41:31Z)
• SearchMaster: Grounded and Regulated Self-Play for Search Agents [57.537666863340284]
  Training search agents requires high-quality search data.<n> Existing pipelines often depend on human-written tasks, expert demonstrations, or stronger teacher models.<n>We present SearchMaster, a self-play framework that trains a single LLM from search tasks it generates, solves, and verifies in a local search environment.
  arXiv  Detail & Related papers  (2026-08-03T07:29:11Z)
• HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning [14.484579581234327]
  We introduce HopRefusalBench, the first benchmark of refusal within multi-hop search.<n>It crosses three causes of unanswerability with root, middle, and terminal topologies.<n>We propose a final-outcome taxonomy spanning target-aware refusal, pseudo-refusal, hallucinated completion, and search-budget exhaustion.
  arXiv  Detail & Related papers  (2026-08-02T16:20:55Z)
• Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories [51.22051230894794]
  Evaluation based on final answers shows whether an agent succeeds, but not which parts of the trajectory make the answer unreliable.<n>We collect 2,790 real trajectories from two agent frameworks, three backbone models, and three benchmarks, convert raw logs into semantic spans, and annotate harmful error spans through expert review.<n>We propose DRIFT, a claim-centric auditing framework that tracks agent claims, checks their support in trajectory evidence, and marks spans where unsupported or conflicting claims affect the answer path.
  arXiv  Detail & Related papers  (2026-06-01T10:50:26Z)
• BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent [74.10138164281618]
  BrowseComp-Plus is a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus.<n>This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods.
  arXiv  Detail & Related papers  (2025-08-08T17:55:11Z)
• SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories [55.161075901665946]
  Super aims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges, and 602 automatically generated problems for larger-scale development. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios.
  arXiv  Detail & Related papers  (2024-09-11T17:37:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.