GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

• URL: http://arxiv.org/abs/2604.15715v1
• Date: Fri, 17 Apr 2026 05:36:00 GMT
• Title: GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows
• Abstract summary: GTA-2 is a hierarchical benchmark for General Tool Agents (GTA)<n>Built on real-world authenticity, it leverages real user queries, deployed tools, and multimodal contexts.<n>Experiments reveal a pronounced capability cliff: while frontier models already struggle on atomic tasks, top models achieving only 14.39% success.
• Score: 90.35728421223673
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The development of general-purpose agents requires a shift from executing simple instructions to completing complex, real-world productivity workflows. However, current tool-use benchmarks remain misaligned with real-world requirements, relying on AI-generated queries, dummy tools, and limited system-level coordination. To address this, we propose GTA-2, a hierarchical benchmark for General Tool Agents (GTA) spanning atomic tool use and open-ended workflows. Built on real-world authenticity, it leverages real user queries, deployed tools, and multimodal contexts. (i) GTA-Atomic, inherited from our prior GTA benchmark, evaluates short-horizon, closed-ended tool-use precision. (ii) GTA-Workflow introduces long-horizon, open-ended tasks for realistic end-to-end completion. To evaluate open-ended deliverables, we propose a recursive checkpoint-based evaluation mechanism that decomposes objectives into verifiable sub-goals, enabling unified evaluation of both model capabilities and agent execution frameworks (i.e., execution harnesses). Experiments reveal a pronounced capability cliff: while frontier models already struggle on atomic tasks (below 50%), they largely fail on workflows, with top models achieving only 14.39% success. Further analysis shows that checkpoint-guided feedback improves performance, while advanced frameworks such as Manus and OpenClaw substantially enhance workflow completion, highlighting the importance of execution harness design beyond the underlying model capacity. These findings provide guidance for developing reliable personal and professional assistants. Dataset and code will be available at https://github.com/open-compass/GTA.

Related papers

• Execution-First Synthetic Tool-Use Trace Generation for LLM Agents [10.291533584425252]
  SyntheticAgentTraceQA is an execution- first framework for generating scalable supervision data for tool- augmented agents.<n>We evaluate the framework across four tool ecosystems and use the resulting data to fine-tune and evaluate Qwen model variants.
  arXiv  Detail & Related papers  (2026-07-31T08:52:26Z)
• Beyond Generalist LLMs: Specialist Agentic Systems for Structured Code Workflow Execution [0.40631409309544836]
  Large Language Models (LLMs) have accelerated the adoption of software development agents, now widely available as Integrated Development Environment (IDE) extensions and standalone applications.<n>While these agents are typically general-purpose, it remains unclear whether specialist agents justify their additional development effort.<n>We investigate this question in the context of business process automation, focusing on the transformation of Business Process Model and Notation (BPMN) diagrams into executable agentic.
  arXiv  Detail & Related papers  (2026-07-16T01:06:38Z)
• AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments [84.8170304874972]
  We present AgentGym2, a new evaluation framework with task instances grounded in real-world working demands.<n>It measures agents' ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information.
  arXiv  Detail & Related papers  (2026-07-06T14:56:51Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• PerfGuard: A Performance-Aware Agent for Visual Content Generation [53.591105729011595]
  PerfGuard is a performance-aware agent framework for visual content generation.<n>It integrates tool performance boundaries into task planning and scheduling.<n>It has advantages in tool selection accuracy, execution reliability, and alignment with user intent.
  arXiv  Detail & Related papers  (2026-01-30T05:12:19Z)
• Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents [52.30603055218294]
  Trajectory2Task is a verifiable data generation pipeline for studying tool use at scale under three realistic user scenarios.<n>It converts valid tool-call trajectories into user-facing tasks with controlled intent adaptations.<n>We benchmark seven state-of-the-art LLMs on the generated complex user scenario tasks and observe frequent failures.
  arXiv  Detail & Related papers  (2026-01-28T00:36:13Z)
• ML-Tool-Bench: Tool-Augmented Planning for ML Tasks [23.54937738755734]
  We introduce a benchmark for evaluating tool-augmented machine learning agents.<n>Our benchmark goes beyond traditional tool-use evaluation by incorporating an in-memory named object management.<n>Our approach improves over ReAct by 16.52 percentile positions, taking the median across all Kaggle challenges.
  arXiv  Detail & Related papers  (2025-11-29T23:59:40Z)
• Multi-Agent Systems for Dataset Adaptation in Software Engineering: Capabilities, Limitations, and Future Directions [8.97512410819274]
  This paper presents the first empirical study on how state-of-the-art multi-agent systems perform in dataset adaptation tasks.<n>We evaluate GitHub Copilot on adapting SE research artifacts from benchmark repositories including ROCODE and LogHub2.0.<n>Results show that current systems can identify key files and generate partial adaptations but rarely produce correct implementations.
  arXiv  Detail & Related papers  (2025-11-26T13:26:11Z)
• Blueprint First, Model Second: A Framework for Deterministic LLM Workflow [3.9886771197662925]
  We introduce the Source Code Agent framework, a new paradigm built on the "Blueprint First, Model Second" philosophy.<n>Our framework decouples the workflow logic from the generative model.<n>Our work enables the verifiable and reliable deployment of autonomous agents in applications governed by strict procedural logic.
  arXiv  Detail & Related papers  (2025-08-01T03:10:00Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• GTA: A Benchmark for General Tool Agents [32.443456248222695]
  We design 229 real-world tasks and executable tool chains to evaluate mainstream large language models (LLMs) Our findings show that real-world user queries are challenging for existing LLMs, with GPT-4 completing less than 50% of the tasks and most LLMs achieving below 25%. This evaluation reveals the bottlenecks in the tool-use capabilities of current LLMs in real-world scenarios, which provides future direction for advancing general-purpose tool agents.
  arXiv  Detail & Related papers  (2024-07-11T17:50:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.