Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?

• URL: http://arxiv.org/abs/2604.03016v1
• Date: Fri, 03 Apr 2026 13:02:01 GMT
• Title: Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?
• Abstract summary: Agentic-MME is a process-verified benchmark for Multimodal Agentic Capabilities.<n>It contains 418 real-world tasks across 6 domains and 3 difficulty levels.<n>It features over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task.
• Score: 35.30497528897595
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) are evolving from passive observers into active agents, solving problems through Visual Expansion (invoking visual tools) and Knowledge Expansion (open-web search). However, existing evaluations fall short: they lack flexible tool integration, test visual and search tools separately, and evaluate primarily by final answers. Consequently, they cannot verify if tools were actually invoked, applied correctly, or used efficiently. To address this, we introduce Agentic-MME, a process-verified benchmark for Multimodal Agentic Capabilities. It contains 418 real-world tasks across 6 domains and 3 difficulty levels to evaluate capability synergy, featuring over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task. Each task includes a unified evaluation framework supporting sandboxed code and APIs, alongside a human reference trajectory annotated with stepwise checkpoints along dual-axis: S-axis and V-axis. To enable true process-level verification, we audit fine-grained intermediate states rather than just final answers, and quantify efficiency via an overthinking metric relative to human trajectories. Experimental results show the best model, Gemini3-pro, achieves 56.3% overall accuracy, which falls significantly to 23.0% on Level-3 tasks, underscoring the difficulty of real-world multimodal agentic problem solving.

Related papers

• RestoreBench: Can AI Agents Restore Power Flow Convergence? [2.4326395001229053]
  Diagnosing and resolving non-convergent power flow cases is a promising yet largely unexplored application.<n>We introduce a benchmark that evaluates these capabilities across multiple LLMs and three architectures.<n>The evaluation covers two power grids and 46 cases per grid, each requiring one or more corrective actions to restore convergence.
  arXiv  Detail & Related papers  (2026-08-31T21:15:19Z)
• AutoMedBench: Towards Medical AutoResearch with Agentic AI Models [31.335309699875264]
  AutoMedBench is a workflow-aware benchmark for autonomous medical-AI research.<n>It comprises long-horizon tasks with each run averaging 33 agent turns.
  arXiv  Detail & Related papers  (2026-06-01T09:22:55Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios [32.58358574768901]
  Real-world multimodal agents solve multi-step grounded in visual evidence.<n>Existing benchmarks mainly evaluate single-turn visual reasoning or specific tool skills.<n>We introduce AgentVista, a benchmark for generalist multimodal agents.
  arXiv  Detail & Related papers  (2026-02-26T16:30:46Z)
• InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents [60.89180545430896]
  InfoMosaic-Bench is the first benchmark dedicated to multi-source information seeking in tool-augmented agents.<n>It requires agents to combine general-purpose search with domain-specific tools.<n>This design guarantees both reliability and non-triviality.
  arXiv  Detail & Related papers  (2025-10-02T17:48:03Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)
• What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities [56.646832992178105]
  We introduce OmniBench, a cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity.<n>We present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities.<n>Our dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate.
  arXiv  Detail & Related papers  (2025-06-10T15:59:38Z)
• Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks [94.19506319646376]
  We introduce Agent-X, a benchmark for evaluating vision-centric agents in real-world, multimodal settings.<n>Agent-X features 828 agentic tasks with authentic visual contexts, including images, multi-image comparisons, videos, and instructional text.<n>Our results reveal that even the best-performing models, including GPT, Gemini, and Qwen families, struggle to solve multi-step vision tasks.
  arXiv  Detail & Related papers  (2025-05-30T17:59:53Z)
• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains [54.117238759317004]
  Massive Multitask Agent Understanding (MMAU) benchmark features comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents.
  arXiv  Detail & Related papers  (2024-07-18T00:58:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.