DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments

• URL: http://arxiv.org/abs/2607.17291v1
• Date: Sun, 19 Jul 2026 15:20:40 GMT
• Title: DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments
• Abstract summary: We introduce DRNOISE, a 100-task benchmark for answer recovery under misleading evidence.<n>Each task has a unique gold answer supported by two corroborating indirect record chains; the paired noisy condition adds one plausible document that states a conflicting answer directly.<n>Across agents with strong clean-task performance, this single intervention causes 66-88 percentage-point accuracy drops.
• Score: 51.049999642138
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep research agents increasingly operate over the open web, where relevant records coexist with redundant summaries, outdated reports, and misleading documents. Existing evaluations offer limited insight into whether agents preserve sound evidential standards when an ordinary-looking false document is deliberately seeded into a searchable environment and offers a direct shortcut to a conflicting answer. We introduce DRNOISE, a 100-task benchmark for answer recovery under misleading evidence. Each task has a unique gold answer supported by two corroborating indirect record chains; the paired noisy condition adds one plausible document that states a conflicting answer directly. The benchmark spans ten families of evidence operations. Across agents with strong clean-task performance, this single intervention causes 66-88 percentage-point accuracy drops. Trace analyses identify verification inertia as the dominant failure mode: agents often retrieve truthful records but stop before completing and reconciling the evidence chain, instead deferring to the answer-like document. Generic verification prompts reduce but do not close this gap. The setting is especially relevant to open-web deployment, where plausible falsehoods arrive through ordinary-looking pages rather than explicit attacks. Reliable deep research therefore requires more than retrieval and citation; it requires active reconciliation of direct claims with record-level evidence.

Related papers

• Navigating Sparse Evidence: Agentic Visual RAG via Explicit Context Selection and Consolidation [50.92223196290564]
  We propose a unified agent loop for explicit evidence selection and consolidation.<n>During exploration, SCoRE retains only query-relevant observations and their source pointers in a maintained textual ledger.<n>At termination, it reloads the referenced original images and consolidates the visual evidence for answering, arranging it into a logical sequence.
  arXiv  Detail & Related papers  (2026-09-14T16:11:35Z)
• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions [29.22067195388263]
  Deep Research agents conduct long-horizon investigations by iteratively planning, retrieving evidence, and generating reports.<n>It remains unclear whether they can resist apparently credible but factually false information introduced into these defenses.<n>We introduce MisKnow-Agent, a framework that constructs task-specific documents supporting manually audited false conclusions.
  arXiv  Detail & Related papers  (2026-07-23T03:28:08Z)
• WILDTRACE: Benchmarking Natural Evidence Trails in Long-Context Reasoning [63.9071746889252]
  Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across distant passages.<n>This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it.<n>We introduce WILDTRACE, a benchmark of 481 tasks over 214 naturally occurring long-form sources.
  arXiv  Detail & Related papers  (2026-07-10T12:09:21Z)
• From Fluent to Verifiable: Claim-Level Auditability for Deep Research Agents [8.49451413641847]
  We argue that as research generation becomes cheap, auditability becomes the bottleneck.<n>This perspective proposes claim-level auditability as a first-class design and evaluation target for deep research agents.
  arXiv  Detail & Related papers  (2026-02-14T19:39:15Z)
• PATHWAYS: Evaluating Investigation and Context Discovery in AI Web Agents [0.0]
  PATHWAYS is a benchmark of 250 multi-step decision tasks.<n>It tests whether web-based agents can discover and correctly use hidden contextual information.
  arXiv  Detail & Related papers  (2026-02-05T06:24:23Z)
• Orchestrating Specialized Agents for Trustworthy Enterprise RAG [8.772844442593975]
  One-pass retrieval-and-write pipelines often yield shallow summaries.<n>We introduce ADORE, an agentic framework that replaces linear retrieval with iterative, user-steered investigation.<n>Our contributions are threefold: Memory-locked synthesis, Evidence-coverage-guided execution, and section-packed long-context grounding.
  arXiv  Detail & Related papers  (2026-01-26T08:48:41Z)
• Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence [56.09494651178128]
  Retrieval models are commonly used in Information Retrieval (IR) applications, such as Retrieval-Augmented Generation (RAG)<n>We quantify the impact of biases, such as a preference for shorter documents, on retrievers like Dragon+ and Contriever.<n>We uncover major vulnerabilities, showing retrievers favor shorter documents, early positions, repeated entities, and literal matches, all while ignoring the answer's presence!
  arXiv  Detail & Related papers  (2025-03-06T23:23:13Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.