Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI

• URL: http://arxiv.org/abs/2607.22368v1
• Date: Fri, 24 Jul 2026 14:55:19 GMT
• Title: Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
• Abstract summary: Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction.<n>Recent reward-hacking benchmarks and system reports show that agents can instead recover public solutions, read evaluation artifacts, infer generator structure, manipulate feedback, or benefit from invalid scoring paths.<n>We formulate protocol validity and introduce HackDetect, a post-hoc audit that identifies an exposure, determines how the agent used it, and assesses whether the resulting score is misleading.
• Score: 6.058290832294911
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction. Their scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success. Recent reward-hacking benchmarks and system reports show that agents can instead recover public solutions, read evaluation artifacts, infer generator structure, manipulate feedback, or benefit from invalid scoring paths; existing responses do not provide a common procedure for attributing these shortcuts and quantifying their effect across benchmarks. We formulate protocol validity and introduce HackDetect, a post-hoc audit that identifies an exposure, determines how the agent used it, and assesses whether the resulting score is misleading. We quantify score inflation with the Mislead gap, defined as the exploit score minus the intended score. We audit 2,385 traces across 15 agent benchmarks and find evidence of exposures and reward hacking in 67.0% of Frontier Science traces and 66.7% of AutoLab tasks. Across paired comparisons, we measure score inflation of 0.45-1.00, showing that benchmark reports should provide evidence that scores reflect the intended capability.

Related papers

• BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure [47.61632934139226]
  LM-agent benchmarks increasingly function as interactive evaluation infrastructure.<n>This interactivity makes evaluations vulnerable to reward hacking.<n>This paper presents BenchShield, a model-backed instrumentation layer for reward integrity.
  arXiv  Detail & Related papers  (2026-09-10T03:10:58Z)
• The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean [2.822987192839875]
  Agent benchmarks are increasingly used to compare large language models (LLMs) and guide deployment decisions.<n>We identify a double measurement confound: execution-critical decisions are performed by a fixed scaffold instead of the model, while the scorer evaluates outputs using criteria that may not reflect task correctness.<n>We unify these issues within a measurement-theoretic framework that characterizes when benchmark scores can be interpreted as evidence of model capability.
  arXiv  Detail & Related papers  (2026-09-06T21:23:11Z)
• Grounded Checklist Partial Credit for Agent Skill Trajectories [5.028537745773042]
  Grounded Checklist Partial Credit (GCPC) is a human-governed partial-credit evaluation of agent trajectories.<n>Human evaluation on 96 trajectories from 12 tasks shows that G CPC aligns more closely with human assessments of progress.<n>The pipeline also transfers to Terminal-Bench and SWE-bench, demonstrating applicability beyond skill-conditioned evaluation.
  arXiv  Detail & Related papers  (2026-08-26T06:17:19Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation [3.079997053226693]
  We present a systematic validity and audit of four major tool-calling benchmark families.<n>Across 496 expert-reviewed benchmark tasks, we find 92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate.<n>We introduce a unified taxonomy of tool-calling evaluation failures, release trace-level audit artifacts and corrected evaluation components, and argue for metrics that separately measure tool invocation, task completion, and outcome verification.
  arXiv  Detail & Related papers  (2026-06-30T21:10:42Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation [8.049844623361725]
  Interactive agent benchmarks map an agent run to a binary outcome through outcome checks.<n>These checks rely on surface level signals or fail to capture the agent's actual action path.<n>We introduce an outcome evidence reporting layer for existing benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T12:20:15Z)
• Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents [6.158612515104146]
  We introduce Helpfulness-Harmfulness metrics to measure the tradeoff between agent feedback and reviewer feedback.<n>We evaluate our approach on BFCL and Tau2-Bench (multi-turn stateful scenarios), achieving +5.5% on irrelevance detection and +7.1% on multi-turn tasks.<n>Our metrics reveal that reviewer model choice is critical: the reasoning model o3-mini achieves a 3:1 benefit-to-risk ratio versus 2.1:1 for GPT-4o.
  arXiv  Detail & Related papers  (2026-04-29T22:09:47Z)
• Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents [0.0]
  NabaOS is a lightweight verification framework for interactive agents.<n>It classifies every claim in an AI response by its source.<n>It detects 94.2% of fabricated tool executions, 87.6% of misstating counts, and 78.4% of false absence claims.
  arXiv  Detail & Related papers  (2026-03-09T20:45:41Z)
• DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning [2.0329381271887255]
  We introduce Dual-agent based Evaluation Through Obscure Under-specified Retrieval (DETOUR), a dual-agent evaluation benchmark containing 1,011 prompts.<n>Our results indicate that current state-of-the-art models still struggle with our benchmark, only achieving 36% accuracy when evaluated on all modalities.
  arXiv  Detail & Related papers  (2026-01-30T22:01:30Z)
• Preventing the Collapse of Peer Review Requires Verification-First AI [49.995126139461085]
  We propose truth-coupling, i.e. how tightly venue scores track latent scientific truth.<n>We formalize two forces that drive a phase transition toward proxy-sovereign evaluation.
  arXiv  Detail & Related papers  (2026-01-23T17:17:32Z)
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection [55.957275374847484]
  VulAgent is a multi-agent vulnerability detection framework based on hypothesis validation.<n>It implements a semantics-sensitive, multi-view detection pipeline, each aligned to a specific analysis perspective.<n>On average, VulAgent improves overall accuracy by 6.6%, increases the correct identification rate of vulnerable--fixed code pairs by up to 450%, and reduces the false positive rate by about 36%.
  arXiv  Detail & Related papers  (2025-09-15T02:25:38Z)
• ScoreGAN: A Fraud Review Detector based on Multi Task Learning of Regulated GAN with Data Augmentation [50.779498955162644]
  We propose ScoreGAN for fraud review detection that makes use of both review text and review rating scores in the generation and detection process. Results show that the proposed framework outperformed the existing state-of-the-art framework, namely FakeGAN, in terms of AP by 7%, and 5% on the Yelp and TripAdvisor datasets.
  arXiv  Detail & Related papers  (2020-06-11T16:15:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.