DR$^{3}$-Eval: Towards Realistic and Reproducible Deep Research Evaluation

• URL: http://arxiv.org/abs/2604.14683v1
• Date: Thu, 16 Apr 2026 06:40:02 GMT
• Title: DR$^{3}$-Eval: Towards Realistic and Reproducible Deep Research Evaluation
• Abstract summary: Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation.<n>We propose DR$3$-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation.
• Score: 55.02569059688697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR$^{3}$-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR$^{3}$-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR$^{3}$-Agent based on multiple state-of-the-art language models demonstrate that DR$^{3}$-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.

Related papers

• DuMate-DeepResearch: An Auditable Multi-Agent System with Recursive Search and Rubric-Grounded Reasoning [58.09293442211248]
  Deep Research (DR) has emerged as a new agentic paradigm to tackle complex, open-ended research tasks.<n>This report presents DuMate-DeepResearch, a multi-agent DR framework built on the Qianfan Agent Foundry.
  arXiv  Detail & Related papers  (2026-06-05T14:10:48Z)
• Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation [74.0621258662676]
  We propose Ptah, a multi-agent harness for interleaved report generation.<n>Ptah orchestrates the lifecycle from user query to rendered web report through planning, research, and writing stages.<n>A verifier agent serves as the harness's acceptance function, enforcing factual grounding, citation fidelity, and cross-modal consistency throughout the workflow.
  arXiv  Detail & Related papers  (2026-05-28T12:40:34Z)
• AgentCPM-Report: Interleaving Drafting and Deepening for Open-Ended Deep Research [85.51475655916026]
  AgentCPM-Report is a lightweight yet high-performing local solution composed of a framework that mirrors the human writing process.<n>Our framework uses a Writing As Reasoning Policy (WARP), which enables models to dynamically revise outlines.<n>Experiments on DeepResearch Bench, DeepConsult, and DeepResearch Gym demonstrate that AgentCPM-Report outperforms leading closed-source systems.
  arXiv  Detail & Related papers  (2026-02-06T09:45:04Z)
• DR-Arena: an Automated Evaluation Framework for Deep Research Agents [35.99095633093855]
  Large Language Models (LLMs) increasingly operate as Deep Research (DR) Agents capable of autonomous investigation and information synthesis.<n>Current benchmarks predominantly rely on static datasets, which suffer from limited task generality, temporal misalignment, and data contamination.<n>We introduce DR-Arena, a fully automated evaluation framework that pushes DR agents to their capability limits through dynamic investigation.
  arXiv  Detail & Related papers  (2026-01-15T15:28:21Z)
• DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation [56.886936435727854]
  DeepResearchEval is an automated framework for deep research task construction and agentic evaluation.<n>For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles.<n>For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.
  arXiv  Detail & Related papers  (2026-01-14T18:38:31Z)
• A Hierarchical Tree-based approach for creating Configurable and Static Deep Research Agent (Static-DRA) [0.0]
  This paper introduces the Static Deep Research Agent (Static-DRA), a novel solution built upon a hierarchical Tree-based static workflow.<n>The core contribution is the integration of two user-tunable parameters, Depth and Breadth, which provide granular control over the research intensity.<n>The agent's architecture, comprising Supervisor, Independent, and Worker agents, facilitates effective multi-hop information retrieval.
  arXiv  Detail & Related papers  (2025-12-03T15:37:13Z)
• ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents [11.666923792025313]
  Deep Research (DR) is an emerging agent application that leverages large language models to address open-ended queries.<n>We introduce ResearchRubrics, a standardized benchmark for DR built with over 2,800+ hours of human labor.<n>We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration.
  arXiv  Detail & Related papers  (2025-11-10T23:07:14Z)
• WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research [73.58638285105971]
  This paper tackles textbfopen-ended deep research (OEDR), a complex challenge where AI agents must synthesize vast web-scale information into insightful reports.<n>We introduce textbfWebWeaver, a novel dual-agent framework that emulates the human research process.<n>Our framework establishes a new state-of-the-art across major OEDR benchmarks, including DeepResearch Bench, DeepConsult, and DeepResearchGym.
  arXiv  Detail & Related papers  (2025-09-16T17:57:21Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines [63.22096609916707]
  Multi-modal Retrieval Augmented Multi-modal Generation (M$2$RAG) is a novel task that enables foundation models to process multi-modal web content.<n>Despite its potential impact, M$2$RAG remains understudied, lacking comprehensive analysis and high-quality data resources.
  arXiv  Detail & Related papers  (2024-11-25T13:20:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.