AllocBench: Measuring Online Tool Allocation Capability in LLM Agents

• URL: http://arxiv.org/abs/2607.23332v2
• Date: Fri, 31 Jul 2026 16:23:16 GMT
• Title: AllocBench: Measuring Online Tool Allocation Capability in LLM Agents
• Abstract summary: We introduce a benchmark that tests whether agents exhibit conscious allocation behavior under a fixed budget.<n>We find that every frontier model we test acts near-optimally in the abstract framing but fails to transfer this ability to script-writing.
• Score: 2.3236525323645285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Creating a reusable tool is an investment: an agent pays a fixed cost now in exchange for the potential of future reuse. Therefore, a user should prefer an agent that creates a small number of highly reusable tools, rather than many one-offs. We introduce a paired benchmark that tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in two contexts: an abstract text-based formulation and a code-construction task. We find that every frontier model we test---Claude Haiku, Claude Opus, GPT-5.4-mini, and GPT-5.6 Sol---acts near-optimally in the abstract framing but fails to transfer this ability to script-writing. Through further experiments, we identify the particular failure modes for each model. Notably, the first three models fail even when the scripts are not evaluated, while GPT-5.6 Sol stays selective under that weaker manipulation and collapses only at full construction. Furthermore, an open-source Qwen model policy-trained for abstract allocation generalizes this ability across held-out lexical variations, but sees no improvement at script allocation. Together, these results establish online tool allocation as a significant capability boundary, even for modern frontier models.

Related papers

• Inverting the Bellman Equation: From $Q$-Values to World Models [57.827849584133425]
  We prove that value-based agents trained on a sufficiently rich set of reward functions implicitly encode a unique and accurate world model.<n>We find that policies trained exclusively on a ttReacher agent's implicit world model are quasi-optimal on out-of-distribution, velocity-based goals despite position-only training.
  arXiv  Detail & Related papers  (2026-06-19T07:26:14Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks [25.713629634281077]
  We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage.<n> TASTE selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through difficulty evolution.<n>Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability.
  arXiv  Detail & Related papers  (2026-05-27T14:45:59Z)
• PRIMA: Operational Patterns for Resilient Multi-Agent Research with Verifiable Identity and Convergent Feedback [0.0]
  PRIMA operates as coordinated multi-agent research systems over multi-hour runs.<n>Main contributions are three operational patterns for surviving failure modes.<n>A Graph Isomorphism case study grounds the architectural claims in a generated artifact.
  arXiv  Detail & Related papers  (2026-05-23T23:27:46Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Terminus-4B: Can a Smaller Model Replace Frontier LLMs at Agentic Execution Tasks? [3.636948650519796]
  Small language model (SLM) can achieve comparable performance to frontier models in the task of agentic terminal execution.<n>We present Terminus-4B, which is a post-trained Qwen3-4B model via Supervised Finetuning (SFT) and Reinforcement Learning (RL)<n>We find that Terminus-4B is able to reduce the token usage of the main agent by up to 30% compared to the No Subagent baseline.
  arXiv  Detail & Related papers  (2026-05-04T22:24:24Z)
• Sponge Tool Attack: Stealthy Denial-of-Efficiency against Tool-Augmented Agentic Reasoning [58.432996881401415]
  Recent work augments large language models (LLMs) with external tools to enable agentic reasoning.<n>We propose Sponge Tool Attack (STA), which disrupts agentic reasoning solely by rewriting the input prompt.<n>STA generates benign-looking prompt rewrites from the original one with high semantic fidelity.
  arXiv  Detail & Related papers  (2026-01-24T19:36:51Z)
• ML-Tool-Bench: Tool-Augmented Planning for ML Tasks [23.54937738755734]
  We introduce a benchmark for evaluating tool-augmented machine learning agents.<n>Our benchmark goes beyond traditional tool-use evaluation by incorporating an in-memory named object management.<n>Our approach improves over ReAct by 16.52 percentile positions, taking the median across all Kaggle challenges.
  arXiv  Detail & Related papers  (2025-11-29T23:59:40Z)
• MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use [72.53177559476704]
  We introduce MCPVerse, a real-world benchmark for evaluating agentic tool use.<n> MCPVerse integrates more than 550 real-world, executable tools to create an unprecedented action space exceeding 140k tokens.<n>We benchmarked the state-of-the-art LLMs across three modes (Oracle, Standard, and Max-Scale)
  arXiv  Detail & Related papers  (2025-08-22T09:47:53Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.