PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents

• URL: http://arxiv.org/abs/2608.24509v1
• Date: Tue, 25 Aug 2026 12:56:08 GMT
• Title: PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents
• Abstract summary: PeakBench is a benchmark of executable multi-tool dependency annotations and measured resource profiles.<n>We show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints.<n>PeakBench is a useful testbed for diagnosing resource-aware agent behavior.
• Score: 79.28372522980153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM agents increasingly solve tasks by invoking multiple tools, where parallel execution is essential for low latency but difficult to manage safely. Existing agent benchmarks primarily evaluate tool selection, argument generation, and end-to-end success under mostly serial execution, largely overlooking valid parallelization and resource-constrained scheduling. This missing scheduling dimension creates a practical failure mode: serial execution is safe but slow, while resource-agnostic parallel execution is fast but prone to avoidable resource overflows. To address this gap, we introduce PeakBench, a benchmark of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles. A central challenge in evaluating such workflows is attribution: failures and inefficiencies may arise from incorrect dependency planning, poor resource-constrained scheduling, or both. PeakBench addresses this challenge with a two-part evaluation framework that disentangles logical planning from physical scheduling, with dedicated metrics for each dimension. Using this framework, we show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints. We further show that exposing resource information can reduce avoidable overflows and improve resource utilization, making PeakBench a useful testbed for diagnosing resource-aware agent behavior. Code is available at https://github.com/Czzzk/Staggering-the-Peaks.

Related papers

• OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulation [41.885817564161975]
  We present OrchBench, a simulation-based benchmark for evaluating multi-agent orchestration plans in isolation.<n>A deterministic simulator evaluates the resulting plan without invoking worker agents and returns interpretable measures of result quality, makespan, and token cost.<n>Across diverse planners and workflow scales, we find that preserving task-critical information is more important than simply increasing the number of agents.
  arXiv  Detail & Related papers  (2026-07-28T12:43:34Z)
• ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning [110.46431027868954]
  ThinkBooster is a framework for seamless test-time compute scaling of large language model (LLM) reasoning.<n>It consists of a modular Python library implementing state-of-the-art TTC scaling strategy and scorer families.<n>A deployable OpenAI-compatible proxy service enables drop-in integration of adaptive reasoning into real-world applications.
  arXiv  Detail & Related papers  (2026-06-05T05:28:46Z)
• Generative Profiling for Soft Real-Time Systems and its Applications to Resource Allocation [1.6495587226184654]
  Real-time systems require accurate characterization of task timing behavior to ensure predictable performance.<n>Existing methods, such as worst-case execution time analysis, often fail to capture the fine-grained timing behaviors of a task under varying resource contexts.<n>We introduce a novel generative profiling approach that synthesizes context-dependent, fine-grained timing profiles for real-time tasks.
  arXiv  Detail & Related papers  (2026-04-01T22:27:27Z)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems [49.78267008828593]
  Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results.<n>Recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories.<n>We propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks.
  arXiv  Detail & Related papers  (2026-02-05T18:20:21Z)
• GAP: Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learning [20.75113227786218]
  Graph-based Agent Planning (GAP) is a novel framework that explicitly models inter-task dependencies through graph-based planning.<n>Our approach trains agent foundation models to decompose complex tasks into dependency-aware sub-task graphs.<n>This dependency-aware orchestration achieves substantial improvements in both execution efficiency and task accuracy.
  arXiv  Detail & Related papers  (2025-10-29T09:35:55Z)
• MSC-Bench: A Rigorous Benchmark for Multi-Server Tool Orchestration [0.0]
  MSC-Bench is a large-scale benchmark for evaluating multi-hop, end-to-end tool orchestration by LLM agents.<n>It addresses gaps by constructing ground truth through 'equal function sets', allowing objective metrics such as F1 score.<n>It systematically tests agent capabilities from single-tool orchestration to complex cross-server planning, and robustness to out-of-scope requests.
  arXiv  Detail & Related papers  (2025-10-22T09:45:11Z)
• Just-in-time Episodic Feedback Hinter: Leveraging Offline Knowledge to Improve LLM Agents Adaptation [77.90555621662345]
  We present JEF Hinter, an agentic system that distills offline traces into compact, context-aware hints.<n>A zooming mechanism highlights decisive steps in long trajectories, capturing both strategies and pitfalls.<n>Experiments on MiniWoB++, WorkArena-L1, and WebArena-Lite show that JEF Hinter consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-10-05T21:34:42Z)
• Semantic-Aware Scheduling for GPU Clusters with Large Language Models [60.14838697778884]
  We propose SchedMate, a framework that bridges the semantic gap between schedulers and jobs they manage.<n>SchedMate extracts deep insights from overlooked, unstructured data sources: source code, runtime logs, and historical jobs.<n>We show SchedMate reduces average job completion times by up to 1.91x, substantially enhancing the scheduling performance.
  arXiv  Detail & Related papers  (2025-10-02T02:01:02Z)
• Haste Makes Waste: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions [56.88110850242265]
  We present Recipe2Plan, a novel benchmark framework based on real-world cooking scenarios.<n>Unlike conventional benchmarks, Recipe2Plan challenges agents to optimize cooking time through parallel task execution.
  arXiv  Detail & Related papers  (2025-03-04T03:27:02Z)
• Scalable and Cost-Efficient ML Inference: Parallel Batch Processing with Serverless Functions [0.36832029288386137]
  This paper explores how serverless architectures can make large-scale ML inference tasks faster and cost-effective.<n>We demonstrate that serverless parallel processing can reduce execution time by over 95% compared to monolithic approaches, at the same cost.
  arXiv  Detail & Related papers  (2025-01-30T15:47:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.