MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents

• URL: http://arxiv.org/abs/2601.12346v1
• Date: Sun, 18 Jan 2026 10:41:33 GMT
• Title: MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents
• Abstract summary: We introduce MMDeepResearch-Bench (MMDR-Bench), a benchmark of 140 expert-crafted tasks across 21 domains.<n>Compared to prior setups, MMDR-Bench emphasizes report-style synthesis with explicit evidence use.<n>We propose a unified, interpretable evaluation pipeline: Formula-LLM Adaptive Evaluation (FLAE) for report quality, Trustworthy Retrieval-Aligned Citation Evaluation (TRACE) for citation-grounded evidence alignment, and Multimodal Support-Aligned Integrity Check (MOSAIC) for text-visual integrity.
• Score: 37.98503734345155
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep Research Agents (DRAs) generate citation-rich reports via multi-step search and synthesis, yet existing benchmarks mainly target text-only settings or short-form multimodal QA, missing end-to-end multimodal evidence use. We introduce MMDeepResearch-Bench (MMDR-Bench), a benchmark of 140 expert-crafted tasks across 21 domains, where each task provides an image-text bundle to evaluate multimodal understanding and citation-grounded report generation. Compared to prior setups, MMDR-Bench emphasizes report-style synthesis with explicit evidence use, where models must connect visual artifacts to sourced claims and maintain consistency across narrative, citations, and visual references. We further propose a unified, interpretable evaluation pipeline: Formula-LLM Adaptive Evaluation (FLAE) for report quality, Trustworthy Retrieval-Aligned Citation Evaluation (TRACE) for citation-grounded evidence alignment, and Multimodal Support-Aligned Integrity Check (MOSAIC) for text-visual integrity, each producing fine-grained signals that support error diagnosis beyond a single overall score. Experiments across 25 state-of-the-art models reveal systematic trade-offs between generation quality, citation discipline, and multimodal grounding, highlighting that strong prose alone does not guarantee faithful evidence use and that multimodal integrity remains a key bottleneck for deep research agents.

Related papers

• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• TVIR: Building Deep Research Agents Towards Text--Visual Interleaved Report Generation [63.89818340842003]
  TVIR (Text--Visual Interleaved Report Generation) is a benchmark of 100 expert-curated multimodal deep research tasks.<n>TVIR-Agent is a hierarchical multi-agent framework that serves as a strong baseline for constructing outlines and retrieving images.<n>We develop a dual-path evaluation framework that combines Textual Assessment and Visual Assessment.
  arXiv  Detail & Related papers  (2026-06-01T14:35:14Z)
• Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation [74.0621258662676]
  We propose Ptah, a multi-agent harness for interleaved report generation.<n>Ptah orchestrates the lifecycle from user query to rendered web report through planning, research, and writing stages.<n>A verifier agent serves as the harness's acceptance function, enforcing factual grounding, citation fidelity, and cross-modal consistency throughout the workflow.
  arXiv  Detail & Related papers  (2026-05-28T12:40:34Z)
• Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation [42.800747398935044]
  Deep-Reporter is a unified agentic framework for grounded multimodal long-form generation.<n>We develop a rigorous curation pipeline producing 8K high-quality agentic traces for model optimization.<n>M2LongBench is a comprehensive testbed comprising 247 research tasks across 9 domains and a stable multimodal sandbox.
  arXiv  Detail & Related papers  (2026-04-12T17:30:44Z)
• Multimodal Fact-Level Attribution for Verifiable Reasoning [80.60864342985748]
  Multimodal large language models (MLLMs) are increasingly used for real-world tasks involving multi-step reasoning and long-form generation.<n>Existing multimodal grounding benchmarks and evaluation methods fail to assess attribution in complex multimodal reasoning.<n>We introduce MuRGAt, a benchmark for evaluating fact-level multimodal attribution in settings that require reasoning beyond direct observation.
  arXiv  Detail & Related papers  (2026-02-12T03:10:02Z)
• HiEAG: Evidence-Augmented Generation for Out-of-Context Misinformation Detection [8.609016163081744]
  We propose HiEAG, a novel Hierarchical Evidence-Augmented Generation framework to refine external consistency checking.<n>Our approach decomposes external consistency checking into a comprehensive engine pipeline, which integrates reranking and rewriting, apart from retrieval.<n>Our approach enables explanation for judgment, and achieves impressive performance with instruction tuning.
  arXiv  Detail & Related papers  (2025-11-18T01:11:48Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents [78.3863007028688]
  MM-BrowseComp is a novel benchmark comprising 224 challenging, hand-crafted questions.<n>These questions often incorporate images in prompts, and crucial information encountered during the search and reasoning process may also be embedded within images or videos on webpages.<n>Our comprehensive evaluation of state-of-the-art models on MM-BrowseComp reveals that even top models like OpenAI o3 with tools achieve only 29.02% accuracy.
  arXiv  Detail & Related papers  (2025-08-14T13:46:47Z)
• Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering [60.062194349648195]
  Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents.<n>Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches.<n>We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains.
  arXiv  Detail & Related papers  (2025-05-22T09:52:57Z)
• MCiteBench: A Multimodal Benchmark for Generating Text with Citations [31.793037002996257]
  Multimodal Large Language Models (MLLMs) have advanced in integrating diverse modalities but frequently suffer from hallucination.<n>Existing work primarily focuses on generating citations for text-only content, leaving the challenges of multimodal scenarios largely unexplored.<n>We introduce MCiteBench, the first benchmark designed to assess the ability of MLLMs to generate text with citations in multimodal contexts.
  arXiv  Detail & Related papers  (2025-03-04T13:12:39Z)
• VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation [100.06122876025063]
  This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings.<n>We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG.
  arXiv  Detail & Related papers  (2024-12-14T06:24:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.