Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation

• URL: http://arxiv.org/abs/2607.02577v1
• Date: Tue, 30 Jun 2026 21:10:42 GMT
• Title: Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation
• Abstract summary: We present a systematic validity and audit of four major tool-calling benchmark families.<n>Across 496 expert-reviewed benchmark tasks, we find 92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate.<n>We introduce a unified taxonomy of tool-calling evaluation failures, release trace-level audit artifacts and corrected evaluation components, and argue for metrics that separately measure tool invocation, task completion, and outcome verification.
• Score: 3.079997053226693
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tool-calling benchmarks are increasingly used to rank language-model agents, yet their scores are often treated as ground truth without validating the evaluators themselves. We present a systematic validity and reproducibility audit of four major tool-calling benchmark families: BFCL v4, τ2-Bench, LiveMCPBench, and MCP-Atlas. Across 496 expert-reviewed benchmark tasks, we find 92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate. The failures are not isolated annotation mistakes: deterministic benchmarks exhibit brittle state matching, trajectory lock-in, incorrect ground truths, substring-based communication failures, and reward-basis misalignment, while LLM-judge benchmarks exhibit rubric drift, hallucinated completion, answer-only scoring, and substantial run-to-run variance. In LiveMCPBench, 23 repeated evaluations of the same setup produce scores ranging from 57.9% to 76.8%, a spread of 18.9 percentage points, large enough to change leaderboard conclusions. These results show that current tool-calling scores can reflect evaluator artifacts rather than agent capability. We introduce a unified taxonomy of tool-calling evaluation failures, release trace-level audit artifacts and corrected evaluation components, and argue for decomposed metrics that separately measure tool invocation, task completion, and outcome verification. Our findings suggest that progress in tool-using agents requires benchmarks whose evaluators are themselves reproducible, auditable, and aligned with human judgments of task success. We further introduce Tool-Veritas, a configurable benchmark that combines deterministic state verification with optional qualitative judging, and Harness Lab, an open-source system for benchmark execution, trace inspection, repeated-run comparison, and evaluator debugging.

Related papers

• The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean [2.822987192839875]
  Agent benchmarks are increasingly used to compare large language models (LLMs) and guide deployment decisions.<n>We identify a double measurement confound: execution-critical decisions are performed by a fixed scaffold instead of the model, while the scorer evaluates outputs using criteria that may not reflect task correctness.<n>We unify these issues within a measurement-theoretic framework that characterizes when benchmark scores can be interpreted as evidence of model capability.
  arXiv  Detail & Related papers  (2026-09-06T21:23:11Z)
• Grounded Checklist Partial Credit for Agent Skill Trajectories [5.028537745773042]
  Grounded Checklist Partial Credit (GCPC) is a human-governed partial-credit evaluation of agent trajectories.<n>Human evaluation on 96 trajectories from 12 tasks shows that G CPC aligns more closely with human assessments of progress.<n>The pipeline also transfers to Terminal-Bench and SWE-bench, demonstrating applicability beyond skill-conditioned evaluation.
  arXiv  Detail & Related papers  (2026-08-26T06:17:19Z)
• Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks [1.671764884922859]
  We evaluate 23 models from six families on four established commonsense benchmarks, four reworked variants, three non-commonsense controls, and eight downstream tasks.<n>We compare model rankings, compute controlled correlations, and use leave-one-family-out cross-validation to assess the criterion validity of commonsense benchmarks.
  arXiv  Detail & Related papers  (2026-08-04T08:49:34Z)
• Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI [6.058290832294911]
  Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction.<n>Recent reward-hacking benchmarks and system reports show that agents can instead recover public solutions, read evaluation artifacts, infer generator structure, manipulate feedback, or benefit from invalid scoring paths.<n>We formulate protocol validity and introduce HackDetect, a post-hoc audit that identifies an exposure, determines how the agent used it, and assesses whether the resulting score is misleading.
  arXiv  Detail & Related papers  (2026-07-24T14:55:19Z)
• Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking [0.0]
  We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems.<n>The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions.<n>The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.
  arXiv  Detail & Related papers  (2026-07-13T19:01:56Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents [4.908588441456849]
  ProcCtrlBench is a benchmark for execution-process evaluation in LLM coding agents.<n>It organizes recurrent execution defects into a reusable ontology covering 11 defect types in 4 categories.<n>It evaluates agent trajectories through standardized process evidence rather than final outcomes alone.
  arXiv  Detail & Related papers  (2026-05-18T08:34:48Z)
• Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone [11.663456969895462]
  Alignment evaluation in machine learning has largely become evaluation of models.<n>This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluation alone.
  arXiv  Detail & Related papers  (2026-05-06T03:28:30Z)
• BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks [26.58983143152204]
  BenchGuard is the first automated auditing framework for task-oriented, execution-based agent benchmarks.<n>It identifies 12 author-confirmed issues in ScienceAgentBench and 83.3% of expert-identified issues on the BIXBench Verified-50 subset.<n>A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated benchmark auditing a practical and valuable complement to human review.
  arXiv  Detail & Related papers  (2026-04-27T19:51:25Z)
• DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality [49.62610727661819]
  Existing fact-checkers are primarily designed for general-domain, factoid-style atomic claims.<n>We propose Audit-then-Score (AtS), where benchmark labels and rationales are explicitly revisable.<n>We instantiate AtS as DeepFact-Bench, a versioned DRR factuality benchmark with auditable rationales.
  arXiv  Detail & Related papers  (2026-03-06T05:05:57Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• Fantastic Bugs and Where to Find Them in AI Benchmarks [28.604919035475188]
  We introduce a framework for systematic benchmark revision that leverages statistical analysis of response patterns to flag potentially invalid questions.<n>Our approach builds on a core assumption commonly used in AI evaluations that the mean score sufficiently summarizes model performance.<n>Across nine widely used benchmarks, our method guides expert review to identify problematic questions with up to 84% precision.
  arXiv  Detail & Related papers  (2025-11-20T22:49:21Z)
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains [97.5573252172065]
  We train a family of Automatic Reasoning Evaluators (FARE) with a simple iterative rejection-sampling supervised finetuning approach.<n>FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators.<n>As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH.
  arXiv  Detail & Related papers  (2025-10-20T17:52:06Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning [63.77667876176978]
  Large language models show improved downstream task interpretability when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness is difficult. We present ROS, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics.
  arXiv  Detail & Related papers  (2022-12-15T15:52:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.