Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation

• URL: http://arxiv.org/abs/2606.05241v1
• Date: Wed, 03 Jun 2026 07:11:36 GMT
• Title: Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation
• Abstract summary: We study Search-Time Contamination (STC) in deep research agent evaluation.<n>We find that STC is widespread and can inflate performance by up to 4%.<n>We advocate contamination-aware practices, including isolated sandboxes, transparent search trajectories, and controlled benchmark access.
• Score: 17.913171310675306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Public benchmarks enable fair and reproducible evaluation of LLM reasoning, but they become fragile for deep research agents that actively search the web during inference. Such agents may retrieve public benchmark metadata, question context, or even ground-truth answers via web search. This gives rise to Search-Time Contamination (STC), where external retrieval bypasses intended reasoning and inflates measured performance. We systematically study STC in deep research agent evaluation. We define three contamination types with increasing severity, namely Benchmark Metadata Leakage, Question-Context Leakage, and Explicit Answer Leakage, and develop detection algorithms to identify them and quantify their impact on agent performance. Evaluating modern deep research agents on six public benchmarks, we find that STC is widespread and can inflate performance by up to 4%. Our findings show that existing evaluations may overestimate true reasoning ability. We therefore advocate contamination-aware practices, including isolated sandboxes, transparent search trajectories, and controlled benchmark access.

Related papers

• Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents [63.2133766267587]
  Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence.<n>We study these questions through a trajectory-level diagnosis of long-horizon search agents.<n>We evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence.
  arXiv  Detail & Related papers  (2026-08-03T08:46:48Z)
• Dont Stop Early: Scalable Enterprise Deep Research with Controlled Information Flow and Evidence-Aware Termination [48.39579693613492]
  Enterprise deep research often fails to produce decision-ready reports due to uneven information coverage, context explosion, and premature stopping.<n>We propose a scalable Enterprise Deep Research architecture to address these failures.
  arXiv  Detail & Related papers  (2026-04-27T20:31:27Z)
• FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
  We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-02-02T23:21:13Z)
• Over-Searching in Search-Augmented Large Language Models [22.821710825732563]
  Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval.<n>Over-searching leads to computational inefficiency and hallucinations by incorporating irrelevant context.<n>Our finding shows: (i) search generally improves answer accuracy on answerable queries but harms abstention on unanswerable ones; (ii) over-searching is more pronounced in complex reasoning models and deep research systems; and (iii) the composition of retrieved evidence is crucial, as the presence of negative evidence improves abstention.
  arXiv  Detail & Related papers  (2026-01-09T03:24:46Z)
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents [63.70653857721785]
  We conduct two in-the-wild experiments to demonstrate the prevalence of low-quality search results and their potential to misguide agent behaviors.<n>To counter this threat, we introduce an automated red-teaming framework that is systematic, scalable, and cost-efficient.
  arXiv  Detail & Related papers  (2025-09-28T07:05:17Z)
• Search-Time Data Contamination [18.94571261664399]
  We identify an analogous issue, search-time contamination (STC), in evaluating search-based LLM agents.<n>We find that HuggingFace, an online platform hosting evaluation datasets, appears among retrieved sources in search based agent logs.<n>We propose best practices for benchmark design and result reporting to address this novel form of leakage.
  arXiv  Detail & Related papers  (2025-08-12T22:52:21Z)
• BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent [74.10138164281618]
  BrowseComp-Plus is a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus.<n>This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods.
  arXiv  Detail & Related papers  (2025-08-08T17:55:11Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research [25.368303145176554]
  DeepResearchGym is an open-source sandbox that combines a search API with a rigorous evaluation protocol for benchmarking deep research systems.<n>The API indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN.<n>It achieves lower latency than popular commercial APIs while ensuring stable document rankings across runs, and is freely available for research use.
  arXiv  Detail & Related papers  (2025-05-25T18:16:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.