How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks

• URL: http://arxiv.org/abs/2607.12338v1
• Date: Tue, 14 Jul 2026 04:36:41 GMT
• Title: How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks
• Abstract summary: A task fraction alone does not show whether a partial run supports the same pairwise conclusion as the completed benchmark.<n>We study this question by replaying completed public task-level records from SWE-bench, AppWorld, and tau-bench.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting. A task fraction alone does not show whether a partial run supports the same pairwise conclusion as the completed benchmark. We study this question by replaying completed public task-level records from SWE-bench, AppWorld, and tau-bench. A partial budget counts as enough only when it supports the completed benchmark's decision, covers required task groups, and leaves no more than a target fraction of comparisons unresolved. The required task fraction varies sharply. At the strict 0 percentage point threshold on a 5 percentage point budget grid, AppWorld first meets all targets at 15 percent, tau-bench at 25 percent, and SWE-bench Verified at 90 percent; SWE-bench Lite does not meet all targets by 95 percent under the primary coverage rule. Partial-evaluation reports should state how much one agent must outperform another, how tasks are selected, what coverage rule is required, what decision rule is used, and how many comparisons may remain unresolved.

Related papers

• xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems [57.62172039072062]
  We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities.<n>The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements.<n>The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points.
  arXiv  Detail & Related papers  (2026-09-07T17:30:58Z)
• ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision [0.0]
  LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete.<n>A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation.<n>We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance.
  arXiv  Detail & Related papers  (2026-08-03T16:22:51Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution [1.733151600403503]
  Large language model (LLM) agents increasingly automate multi-step engineering and informatics.<n>They often follow a maximum-context-first strategy--re-reading files and dependencies they have already seen--turning a one-line edit into a small code-base audit.<n>We argue the missing capability is task-aware execution-scope estimation.
  arXiv  Detail & Related papers  (2026-07-14T17:59:31Z)
• DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks [1.7623000586936592]
  DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.<n>Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream.<n>Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code.
  arXiv  Detail & Related papers  (2026-07-08T21:45:34Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents? [15.202813683786337]
  Repository-level performance-optimization benchmarks evaluate coding agents by applying patches to real repositories.<n>We show that leaderboard scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission.<n>We find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450)
  arXiv  Detail & Related papers  (2026-07-01T17:50:48Z)
• SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge? [26.684941358964704]
  SEAL is a self-improving evaluation protocol for extracting latent ranking signal from saturated benchmarks.<n>We evaluate SEAL on multiple saturated benchmarks covering code generation, mathematical reasoning, knowledge-intensive question answering, and tool-use agent task completion.
  arXiv  Detail & Related papers  (2026-05-28T15:46:54Z)
• ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads? [0.8749675983608171]
  We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference tasks.<n>We curated 54 tasks from merged pull requests with measurable performance improvements.
  arXiv  Detail & Related papers  (2026-02-23T08:37:53Z)
• Establishing Best Practices for Building Rigorous Agentic Benchmarks [94.69724201080155]
  We show that many agentic benchmarks have issues in task setup or reward design.<n>Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms.<n>We introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience.
  arXiv  Detail & Related papers  (2025-07-03T17:35:31Z)
• AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories [61.38499597241457]
  We propose AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents.<n>Using our benchmark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks.<n>We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents.
  arXiv  Detail & Related papers  (2025-04-11T19:49:22Z)
• Localizing Task Information for Improved Model Merging and Compression [61.16012721460561]
  We show that the information required to solve each task is still preserved after merging as different tasks mostly use non-overlapping sets of weights. We propose Consensus Merging, an algorithm that eliminates such weights and improves the general performance of existing model merging approaches.
  arXiv  Detail & Related papers  (2024-05-13T14:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.