MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

• URL: http://arxiv.org/abs/2603.28407v1
• Date: Mon, 30 Mar 2026 13:16:03 GMT
• Title: MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome
• Abstract summary: MiroEval is a benchmark and evaluation framework for deep research systems.<n>The benchmark comprises 100 tasks grounded in real user needs.<n>The proposed evaluation suite assesses deep research systems along three complementary dimensions.
• Score: 109.15093810810214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.

Related papers

• Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents [72.19038675107662]
  Mr. L is a benchmark for evaluating real-world deep research over long, irreducible chains of verifiable evidence.<n>Each question is constructed from a hidden Node-Relation graph and requires an average of 12.1 necessary intermediate conclusions.
  arXiv  Detail & Related papers  (2026-09-10T09:47:44Z)
• WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation [26.889929086766003]
  Web agents increasingly demonstrate capabilities in automated task execution.<n>Existing benchmarks suffer from insufficient scale and limited domain diversity.<n>We introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks.
  arXiv  Detail & Related papers  (2026-07-07T10:27:31Z)
• DuMate-DeepResearch: An Auditable Multi-Agent System with Recursive Search and Rubric-Grounded Reasoning [58.09293442211248]
  Deep Research (DR) has emerged as a new agentic paradigm to tackle complex, open-ended research tasks.<n>This report presents DuMate-DeepResearch, a multi-agent DR framework built on the Qianfan Agent Foundry.
  arXiv  Detail & Related papers  (2026-06-05T14:10:48Z)
• FAB-Bench: A Framework for Adaptive RAG Benchmarking in Semiconductor Manufacturing [21.340021473068663]
  We introduce FAB-Bench, an end-to-end framework for adaptive benchmarking of RAG systems in semiconductor manufacturing.<n> FAB-Bench defines six diagnostic metrics measuring factual accuracy, contextual utilization, completeness, retrieval relevance, technical depth, and reasoning consistency.<n>From over 1,300 generated candidates, we curated a high-quality benchmark of 200 query-answer pairs spanning three synthesis strategies.
  arXiv  Detail & Related papers  (2026-05-26T02:32:51Z)
• DR$^{3}$-Eval: Towards Realistic and Reproducible Deep Research Evaluation [55.02569059688697]
  Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation.<n>We propose DR$3$-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation.
  arXiv  Detail & Related papers  (2026-04-16T06:40:02Z)
• LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild [86.6586720134927]
  LiveResearchBench is a benchmark of 100 expert-curated tasks spanning daily life, enterprise, and academia.<n>DeepEval is a comprehensive suite covering both content- and report-level quality.<n>Our analysis reveals current strengths, recurring failure modes, and key system components needed to advance reliable, insightful deep research.
  arXiv  Detail & Related papers  (2025-10-16T02:49:16Z)
• Understanding DeepResearch via Reports [41.60038455664918]
  DeepResearch is a transformative AI paradigm, conducting expert-level research through sophisticated reasoning and multi-tool integration.<n> evaluating these systems remains critically challenging due to open-ended research scenarios and existing benchmarks that focus on isolated capabilities.<n>We introduce DeepResearch-ReportEval, a comprehensive framework designed to assess DeepResearch systems through their most representative outputs: research reports.
  arXiv  Detail & Related papers  (2025-10-09T07:03:43Z)
• A Rigorous Benchmark with Multidimensional Evaluation for Deep Research Agents: From Answers to Reports [24.09178055088843]
  Deep Research Agents (DRAs) exhibit the capabilities for task decomposition, cross-source retrieval, multi-stage reasoning, and structured output.<n>This paper introduces a rigorous benchmark and a multidimensional evaluation framework tailored to DRAs and report-style responses.<n>The framework enables comprehensive evaluation of long-form reports generated by DRAs, incorporating integrated scoring metrics for semantic quality, topical focus, and retrieval trustworthiness.
  arXiv  Detail & Related papers  (2025-10-02T16:40:02Z)
• CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity [23.48167670445722]
  Retrieval-Augmented Generation (RAG) aims to generate more accurate and reliable answers with the help of the retrieved context from external knowledge sources. evaluating these systems remains a crucial research area due to the following issues. We propose a Comprehensive Full-chain Evaluation (CoFE-RAG) framework to facilitate thorough evaluation across the entire RAG pipeline.
  arXiv  Detail & Related papers  (2024-10-16T05:20:32Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs)<n>MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts.<n>It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.