Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

• URL: http://arxiv.org/abs/2607.01211v1
• Date: Wed, 01 Jul 2026 17:50:48 GMT
• Title: Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
• Abstract summary: Repository-level performance-optimization benchmarks evaluate coding agents by applying patches to real repositories.<n>We show that leaderboard scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission.<n>We find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450)
• Score: 15.202813683786337
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.

Related papers

• RobustSGPO: Search-Space Control for Agent Harness Evolution [56.722805974785125]
  We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots.<n>We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts.
  arXiv  Detail & Related papers  (2026-09-09T03:00:03Z)
• Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents [29.96375986740068]
  Harness-IF scores operational rules one at a time from execution evidence.<n>AP-Acc scores only rules labeled as opposing unprompted defaults.
  arXiv  Detail & Related papers  (2026-08-12T07:07:57Z)
• Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks [1.671764884922859]
  We evaluate 23 models from six families on four established commonsense benchmarks, four reworked variants, three non-commonsense controls, and eight downstream tasks.<n>We compare model rankings, compute controlled correlations, and use leave-one-family-out cross-validation to assess the criterion validity of commonsense benchmarks.
  arXiv  Detail & Related papers  (2026-08-04T08:49:34Z)
• DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks [1.7623000586936592]
  DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.<n>Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream.<n>Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code.
  arXiv  Detail & Related papers  (2026-07-08T21:45:34Z)
• Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation [3.079997053226693]
  We present a systematic validity and audit of four major tool-calling benchmark families.<n>Across 496 expert-reviewed benchmark tasks, we find 92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate.<n>We introduce a unified taxonomy of tool-calling evaluation failures, release trace-level audit artifacts and corrected evaluation components, and argue for metrics that separately measure tool invocation, task completion, and outcome verification.
  arXiv  Detail & Related papers  (2026-06-30T21:10:42Z)
• Auditing Reward Hackability in Code RL Training Environments [0.0]
  We measure the rate at which code RL environments accept incorrect solutions as correct.<n>On a 49-task sample of SWE-bench Verified, 28.5% of tasks have test suites weak enough that a Docker-verified incorrect patch passes them.
  arXiv  Detail & Related papers  (2026-06-14T23:31:42Z)
• Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation [19.980368400366352]
  We model benchmarking as a multitask principal-agent game and show that the welfare loss from a benchmark is determined jointly by three item-level primitives.<n>We apply the theory to OLMES items using WORKBank for welfare, the EvoLM 4B suite for improvability, and the PolyPythias 410M panel for variance.
  arXiv  Detail & Related papers  (2026-05-29T07:01:38Z)
• SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge? [26.684941358964704]
  SEAL is a self-improving evaluation protocol for extracting latent ranking signal from saturated benchmarks.<n>We evaluate SEAL on multiple saturated benchmarks covering code generation, mathematical reasoning, knowledge-intensive question answering, and tool-use agent task completion.
  arXiv  Detail & Related papers  (2026-05-28T15:46:54Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)
• Fantastic Bugs and Where to Find Them in AI Benchmarks [28.604919035475188]
  We introduce a framework for systematic benchmark revision that leverages statistical analysis of response patterns to flag potentially invalid questions.<n>Our approach builds on a core assumption commonly used in AI evaluations that the mean score sufficiently summarizes model performance.<n>Across nine widely used benchmarks, our method guides expert review to identify problematic questions with up to 84% precision.
  arXiv  Detail & Related papers  (2025-11-20T22:49:21Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• How Should We Build A Benchmark? Revisiting 274 Code-Related Benchmarks For LLMs [60.25940747590386]
  We propose How2Bench, which is comprised of a 55-criteria checklist as a set of guidelines to govern the development of code-related benchmarks comprehensively.<n>We profiled 274 benchmarks released within the past decade and found concerning issues.<n>Nearly 70% of the benchmarks did not take measures for data quality assurance; over 10% did not even open source or only partially open source.
  arXiv  Detail & Related papers  (2025-01-18T09:51:57Z)
• metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models [5.972993094932516]
  Large Language Models (LLMs) vary in their abilities on a range of tasks.<n>We show that there exists a small set of common underlying abilities that these benchmarks measure.<n>We distill a sparse benchmark, metabench, that has less than 3% of the original size of all six benchmarks combined.
  arXiv  Detail & Related papers  (2024-07-04T17:57:38Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.