Efficient Benchmarking of AI Agents

• URL: http://arxiv.org/abs/2603.23749v1
• Date: Tue, 24 Mar 2026 22:17:11 GMT
• Title: Efficient Benchmarking of AI Agents
• Abstract summary: We study whether small task subsets can preserve agent rankings at substantially lower cost.<n>We find that absolute score prediction degrades under scaffold-driven distribution shift.<n>We propose an optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating AI agents on comprehensive benchmarks is expensive because each evaluation requires interactive rollouts with tool use and multi-step reasoning. We study whether small task subsets can preserve agent rankings at substantially lower cost. Unlike static language model benchmarks, agent evaluation is subject to scaffold-driven distribution shift, since performance depends on the framework wrapping the underlying model. Across eight benchmarks, 33 agent scaffolds, and 70+ model configurations, we find that absolute score prediction degrades under this shift, while rank-order prediction remains stable. Exploiting this asymmetry, we propose a simple optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates (30-70%). This mid-range difficulty filter, motivated by Item Response Theory, reduces the number of evaluation tasks by 44-70% while maintaining high rank fidelity under scaffold and temporal shifts. It provides more reliable rankings than random sampling, which exhibits high variance across seeds, and outperforms greedy task selection under distribution shift. These results suggest that reliable leaderboard ranking does not require full-benchmark evaluation.

Related papers

• Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation [141.29138788636013]
  We develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents.<n>We conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks.<n>We introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T16:26:20Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents [19.290397279235084]
  This paper aggregates the largest coordinated deep-dive of one MCP-based industrial-agent benchmark to date.<n>We argue that aggregate-score leaderboards systematically underspecify deployed-agent evaluation.
  arXiv  Detail & Related papers  (2026-06-18T02:02:46Z)
• Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons [16.73008463123115]
  Low-rank sharing improves sample efficiency over independent task-wise Bradley-Terry estimation.<n>Experiments on synthetic data and Arena show that low-rank sharing improves sample efficiency over independent task-wise Bradley-Terry estimation.
  arXiv  Detail & Related papers  (2026-05-28T05:44:43Z)
• An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation [50.09718257952108]
  Sequentialsolv has increasingly shifted toward generative recommenders that combine sequential patterns with semantic item information.<n>These methods are often evaluated on a small set of widely used benchmarks, raising a key question: do these benchmarks actually require the advanced modeling capabilities that modern generative recommenders claim to provide?<n>We conduct a benchmark audit with an intentionally simple graph. Starting from only the last one or two items, it retrieves candidates from a few-hop item-transition graph and ranks them by item-feature similarity.
  arXiv  Detail & Related papers  (2026-05-08T02:00:11Z)
• A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
  Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
  arXiv  Detail & Related papers  (2026-03-04T07:23:54Z)
• Active Evaluation of General Agents: Problem Definition and Comparison of Baseline Algorithms [18.53965204068826]
  We propose a formal definition and a conceptual framework for active evaluation of agents across multiple tasks.<n>On every iteration, the ranking algorithm chooses the task and agents to sample scores from.<n>We find that the classical Elo rating system is a consistently reliable choice for efficient reduction of ranking error in practice.
  arXiv  Detail & Related papers  (2026-01-12T15:32:11Z)
• Establishing Best Practices for Building Rigorous Agentic Benchmarks [94.69724201080155]
  We show that many agentic benchmarks have issues in task setup or reward design.<n>Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms.<n>We introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience.
  arXiv  Detail & Related papers  (2025-07-03T17:35:31Z)
• SCAN: Structured Capability Assessment and Navigation for LLMs [54.54085382131134]
  textbfSCAN (Structured Capability Assessment and Navigation) is a practical framework that enables detailed characterization of Large Language Models.<n>SCAN incorporates four key components:.<n>TaxBuilder, which extracts capability-indicating tags from queries to construct a hierarchical taxonomy;.<n>RealMix, a query synthesis and filtering mechanism that ensures sufficient evaluation data for each capability tag;.<n>A PC$2$-based (Pre-Comparison-derived Criteria) LLM-as-a-Judge approach achieves significantly higher accuracy compared to classic LLM-as-a-Judge method
  arXiv  Detail & Related papers  (2025-05-10T16:52:40Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Unsupervised Summarization Re-ranking [27.8646077587359]
  In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEG by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks.
  arXiv  Detail & Related papers  (2022-12-19T16:29:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.