The Amazing Agent Race: Strong Tool Users, Weak Navigators

• URL: http://arxiv.org/abs/2604.10261v2
• Date: Fri, 17 Apr 2026 01:52:26 GMT
• Title: The Amazing Agent Race: Strong Tool Users, Weak Navigators
• Abstract summary: Existing tool-use benchmarks for LLM agents are overwhelmingly linear.<n>We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph puzzles (or "legs") with fork-merge tool chains.<n>We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs)
• Score: 27.24330141815882
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of 2 to 5 steps. We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph (DAG) puzzles (or "legs") with fork-merge tool chains. We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs). Agents must navigate Wikipedia, execute multi-step tool chains, and aggregate results into a verifiable answer. Legs are procedurally generated from Wikipedia seeds across four difficulty levels with live-API validation. Three complementary metrics (finish-line accuracy, pit-stop visit rate, and roadblock completion rate) separately diagnose navigation, tool-use, and arithmetic failures. Evaluating three agent frameworks on 1,400 legs, the best achieves only 37.2% accuracy. Navigation errors dominate (27 to 52% of trials) while tool-use errors remain below 17%, and agent architecture matters as much as model scale (Claude Code matches Codex CLI at 37% with 6x fewer tokens). The compositional structure of AAR reveals that agents fail not at calling tools but at navigating to the right pages, a blind spot invisible to linear benchmarks. The project page can be accessed at: https://minnesotanlp.github.io/the-amazing-agent-race

Related papers

• An Experimental Evaluation of Multimodal Prompt Injection Attacks on Agentic AI Frameworks [0.0]
  We present MMPIBench, a reproducible benchmark that measures what happens next.<n>It delivers a fixed set of attacks through six visual carriers (OCR text, overlays, EXIF metadata, QR codes, fake interfaces, and hybrids)<n>Attacks complete in approximately 1% of runs but are attempted in 12.8%, and the gap is closed almost entirely at the planning step.
  arXiv  Detail & Related papers  (2026-09-08T20:00:03Z)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents [69.07066297088187]
  SMITH (-grounded Multi-task Iterative Tool Honing) trains tool creation and tool use inside a single policy.<n>A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks.<n>It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline)
  arXiv  Detail & Related papers  (2026-08-25T13:59:34Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers [0.0]
  We present DynamicMCPBench, a reusable framework rather than a fixed dataset.<n>It generates realistic goals, pursues each one live to record a successful trajectory, distills that trajectory into path-agnostic effect checkpoints, and scores an agent on whether it reproduces those effects, never on the final answer.<n>Even the strongest agents solve only about half of the tasks, 31% of tasks are solved by no model at all, and accuracy collapses as the required tool chain grows longer.
  arXiv  Detail & Related papers  (2026-07-10T12:25:57Z)
• PairCoder++: Pair Programming as a Universal Paradigm for Verified Code-Driven Multimodal and Structured-Artifact Generation [51.92442051257354]
  PairCoder improves essentially every benchmark whose artifact is verifiable, on full official metric suites rather than execution alone.<n>TikZ compile rate up 10 to 30 points on every model, at 2.9 to 9.2 times single model cost.
  arXiv  Detail & Related papers  (2026-07-02T08:36:02Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer [15.00724508326096]
  textbfLLM-as-a-Developer replaces human developers with an LLM coding agent that learns each framework's API from documentation, writes agent code, and iteratively repairs it through a validate-and-feedback loop until tests pass.<n>We implement this in textbfADK Arena, a fully automated pipeline with per-framework Docker isolation, a three-level validation pipeline, and benchmark adapters for SWE-bench, $2$-bench, Terminal-Bench, and MCP-Atlas.
  arXiv  Detail & Related papers  (2026-06-04T01:00:54Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend [2.9612444540570113]
  We present CUJBench, the first benchmark to combine browser-visible failure evidence with backend observability in a diagnostic framing.<n>The benchmark yields an overall accuracy of 19.7% with a ceiling of 52%, well below saturation.
  arXiv  Detail & Related papers  (2026-04-25T22:10:53Z)
• Formal Architecture Descriptors as Navigation Primitives for AI Coding Agents [0.0]
  We investigate whether providing agents with formal architecture descriptors can reduce this navigational overhead.<n>We propose intent.lisp, an S-expression architecture descriptor, and open-source the Forge toolkit.
  arXiv  Detail & Related papers  (2026-04-11T00:26:31Z)
• ClawBench: Can AI Agents Complete Everyday Online Tasks? [50.958690494341106]
  ClawBench is an evaluation framework of 153 simple tasks that people need to accomplish regularly in their lives and work.<n>ClawBench operates on production websites, preserving the full complexity, dynamic nature, and challenges of real-world web interaction.<n>A lightweight interception layer captures and blocks only the final submission request, ensuring safe evaluation without real-world side effects.
  arXiv  Detail & Related papers  (2026-04-09T17:57:13Z)
• ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces [17.202580606345666]
  Large language model (LLM) agents are increasingly deployed to automate productivity tasks.<n>Existing benchmarks rely on simplified environments and fail to capture realistic, multi-service environments.<n>We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings.
  arXiv  Detail & Related papers  (2026-04-06T21:09:06Z)
• APEX-EM: Non-Parametric Online Learning for Autonomous Agents via Structured Procedural-Episodic Experience Replay [7.370176470430802]
  LLM-based autonomous agents lack persistent procedural memory.<n>We present APEX-EM, a non-parametric online learning framework that accumulates, retrieves, and reuses structured procedural plans.
  arXiv  Detail & Related papers  (2026-03-31T00:24:56Z)
• TRAJEVAL: Decomposing Code Agent Trajectories for Fine-Grained Diagnosis [23.834704102474927]
  Code agents can resolve GitHub issues, yet when they fail, current evaluation provides no visibility into where or why.<n>We introduce TRAJEVAL, a diagnostic framework that decomposes agent trajectories into three interpretable stages.<n>We validate that these diagnostics are predictive, achieving model-level Pass@1 prediction within 0.87-2.1% MAE.
  arXiv  Detail & Related papers  (2026-03-25T05:27:03Z)
• CodeCompass: Navigating the Navigation Paradox in Agentic Code Intelligence [0.0]
  We identify the Navigation Paradox: agents perform poorly because navigation and retrieval are fundamentally distinct problems.<n>We demonstrate that graph-based structural navigation via Code--a Model Context Protocol server exposing dependency graphs--achieves 99.4% task completion on hidden-dependency tasks.
  arXiv  Detail & Related papers  (2026-02-23T16:58:37Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution [86.4588675093384]
  Toolathlon is a benchmark for language agents offering diverse Apps and tools, realistic environment setup, and reliable execution-based evaluation.<n>This benchmark includes 108 manually sourced or crafted tasks, requiring interacting with multiple Apps over around 20 turns on average to complete.<n>We expect Toolathlon to drive the development of more capable language agents for real-world, long-horizon task execution.
  arXiv  Detail & Related papers  (2025-10-29T17:32:49Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• NaviAgent: Bilevel Planning on Tool Navigation Graph for Large-Scale Orchestration [13.925896302382043]
  Large language models (LLMs) have recently demonstrated the ability to act as function call agents by invoking external tools.<n>We propose NaviAgent, which decouples task planning from tool execution through graph-based modeling of the tool ecosystem.<n> Experiments show that NaviAgent achieves the best task success rates across models and tasks, and integrating TWMN further boosts performance by up to 17 points on complex tasks.
  arXiv  Detail & Related papers  (2025-06-24T10:39:07Z)
• InSTA: Towards Internet-Scale Training For Agents [49.763517682308766]
  We develop a pipeline to facilitate internet-scale training for agents without laborious human annotations.<n>We train agents based on Qwen 3 1.7B that are competitive with frontier LLMs as web agents, while being smaller and faster.<n>Our top agent reaches a success rate of 56.9%, outperforming the data collection policy Qwen 3B, a 235 times larger Llama 4 Maverick, and reaching 94.7% of the performance of Gemini 2.5 Flash.
  arXiv  Detail & Related papers  (2025-02-10T18:54:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.