TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agents

• URL: http://arxiv.org/abs/2605.16909v1
• Date: Sat, 16 May 2026 09:49:25 GMT
• Title: TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agents
• Abstract summary: MM-ToolBench is a benchmark and evaluation harness for task-oriented omni-modal tool use.<n>The central design of MM-ToolBench is closed-loop multimodal verification.<n>Experiments on 15 contemporary agentic models show that MM-ToolBench remains highly challenging.
• Score: 13.027789222320807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tool-using agents are increasingly expected to operate across realistic professional workflows, where they must interpret multimodal inputs, coordinate external tools, inspect intermediate artifacts, and revise their actions before producing a final result. Existing benchmarks, however, often evaluate tool use, computer use, and multimodal reasoning in isolation, leaving a gap between benchmark settings and end-to-end omni-modal tool use in the real world. To address this gap, we introduce MM-ToolBench, a benchmark and evaluation harness for task-oriented omni-modal tool use. MM-ToolBench contains 100 executable tasks from two macro task families, Customer Service and Intelligent Creation, covering 20 subcategory slices and supported by 27 MCP servers with 324 tools. The central design of MM-ToolBench is closed-loop multimodal verification: agents must execute tools, inspect rendered or transformed artifacts, and self-correct when outputs fail task-specific requirements. To make such evaluation scalable and verifiable, MM-ToolBench couples MCP-based execution with task-specific grounded evaluators and a semi-automated construction pipeline for scenario discovery, task instantiation, evaluator synthesis, and human audit. Experiments on 15 contemporary agentic models show that MM-ToolBench remains highly challenging: Claude Opus 4.6, commonly regarded as one of the strongest coding-agent models, achieves only 32.0% task success, far below the 94.0% human benchmark. We envision MM-ToolBench as a practical foundation for evaluating and advancing next-generation omni-modal tool-using agents through closed-loop multimodal verification.

Related papers

• ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents [0.0]
  ToolFailBench is a diagnostic benchmark for measuring tool-use failures.<n>We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use.
  arXiv  Detail & Related papers  (2026-07-06T05:25:21Z)
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents [52.33697136670754]
  We introduce textbfHyperTool, a unified executable MCP-style tool interface that changes the model-visible unit of tool execution.<n>A model invokes HyperTool with a code block that can call existing tools through their original schemas, manipulate returned values, and pass intermediate results locally.<n>On MCP-Universe, HyperTool improves average accuracy from 15.69% to 35.29% on Qwen3-32B and from 9.93% to 33.33% on Qwen3-8B, and surpass GPT-OSS and Kimi-k2.5 on average
  arXiv  Detail & Related papers  (2026-06-11T17:56:36Z)
• ToolOmni: Enabling Open-World Tool Use via Agentic learning with Proactive Retrieval and Grounded Execution [49.30293260493709]
  Large Language Models (LLMs) enhance their problem-solving capability by utilizing external tools.<n>Existing methods relying on static embedding retrieval or parameter memorization struggle to align user intent with tool semantics.<n>We present Tool Omni, a unified agentic framework that enables LLMs for open-world tool use by proactive retrieval and grounded execution.
  arXiv  Detail & Related papers  (2026-04-15T12:26:10Z)
• Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents [22.64138018985385]
  We argue that failures stem from both tool-use accuracy (how well an agent invokes a tool) and intrinsic tool accuracy (the tool's own correctness)<n>We introduce OpenTools, a community-driven toolbox that standardizes tool schemas.<n>OpenTools includes the core framework, an initial tool set, evaluation pipelines, and a contribution protocol.
  arXiv  Detail & Related papers  (2026-03-31T18:42:36Z)
• VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining [49.78591189918702]
  We introduce VisualToolChain-Bench (VTC-Bench) to evaluate tool-use proficiency in Multimodal Large Language Models (MLLMs)<n>To align with realistic computer vision pipelines, our framework features 32 diverse OpenCV-based visual operations.<n>For precise evaluation, we provide 680 curated problems structured across a nine-category cognitive hierarchy.
  arXiv  Detail & Related papers  (2026-03-16T09:31:44Z)
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers [5.463884405989425]
  We introduce MCP-Atlas, a large-scale benchmark for evaluating tool-use competency.<n>It includes 1,000 tasks designed to assess tool-use competency in realistic, multi-step orchestrate.<n>We score tasks using a claims-based rubric that awards partial credit based on the factual claims satisfied in the model's final answer.
  arXiv  Detail & Related papers  (2026-01-31T23:19:39Z)
• MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use [12.220519951554133]
  MCPAgentBench is a benchmark based on real-world MCP definitions to evaluate the tool-use capabilities of agents.<n>The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors.<n>Experiments conducted on various latest mainstream Large Language Models reveal significant performance differences in handling complex, multi-step tool invocations.
  arXiv  Detail & Related papers  (2025-12-31T02:09:48Z)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents [49.34040731113563]
  We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities.<n> Rigorous manual validation yields 158 high-quality tools, each verified for correct functionality, practical applicability, and versatility.<n> OSWorld-MCP deepens understanding of multimodal agents and sets a new standard for evaluating performance in complex, tool-assisted environments.
  arXiv  Detail & Related papers  (2025-10-28T15:56:36Z)
• MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers [24.6512259539754]
  MCP-Bench is a benchmark for evaluating large language models (LLMs) on realistic, multi-step tasks.<n>Built on the Model Context Protocol (MCP), MCP-Bench connects LLMs to 28 representative live MCP servers spanning 250 tools across domains such as finance, traveling, scientific computing, and academic search.
  arXiv  Detail & Related papers  (2025-08-28T05:58:57Z)
• MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use [72.53177559476704]
  We introduce MCPVerse, a real-world benchmark for evaluating agentic tool use.<n> MCPVerse integrates more than 550 real-world, executable tools to create an unprecedented action space exceeding 140k tokens.<n>We benchmarked the state-of-the-art LLMs across three modes (Oracle, Standard, and Max-Scale)
  arXiv  Detail & Related papers  (2025-08-22T09:47:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.