DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments
Abstract Overview
DRNoise is a 100-task benchmark for evaluating whether deep research agents can recover correct answers when a searchable environment includes a plausible but misleading document. Each task has a unique gold answer supported by two corroborating indirect evidence chains, while the noisy condition adds a single ordinary-looking document that directly states a conflicting answer. The benchmark covers ten evidence-operation families and is designed to isolate how agents change their evidence-use behavior when an answer-like shortcut is introduced. The paper evaluates several contemporary search agents under matched clean and noisy conditions and analyzes their retrieval traces and verification behavior.
Novelty
The paper’s main novelty is a paired clean–noisy benchmark that differs by only one injected false summary, allowing the authors to isolate noise-induced changes in evidence use rather than general task difficulty. It also introduces conditional deference as a targeted metric for measuring when agents abandon previously demonstrated clean-task competence and adopt the designated false answer.
Results
Across agents with strong clean-task performance, adding one misleading document causes large accuracy drops of 66–88 percentage points, with high conditional deference to the false answer. Trace and ablation analyses indicate that the dominant failure mode is premature stopping and insufficient verification: agents often retrieve truthful records but do not complete or reconcile the evidence chain. Generic verification prompting improves noisy accuracy only partially, while structural interventions such as providing direct true evidence or full evidence context reduce but do not eliminate the problem.
Key Points
- DRNoise contains 100 tasks across 10 reasoning families, with each noisy instance adding a single plausible document that directly states a conflicting answer while truthful records remain available.
- The benchmark shows that strong clean-task accuracy does not imply robustness under misleading evidence, as several agents sharply degrade and frequently defer to the injected false value.
- Analyses suggest the main weakness is verification inertia rather than pure retrieval failure, because agents often co-retrieve truthful and false evidence but stop before fully reconciling the record-level support.