Rethinking Code Performance Benchmarks for LLMs

• URL: http://arxiv.org/abs/2607.07619v1
• Date: Wed, 08 Jul 2026 16:36:03 GMT
• Title: Rethinking Code Performance Benchmarks for LLMs
• Abstract summary: We revisit four popular benchmarks: EffiBench, Enamel, EvalPerf, and Mercury.<n>With the benchmark-provided test suites, only 6.11% of the performant implementations are significantly faster than the canonical solutions.<n>We propose an LLM-based multi-agent framework to generate performance-oriented tests that expose runtime differences more effectively than the original tests.
• Score: 21.32071427130668
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many function-level performance benchmarks have been proposed to evaluate whether large language models (LLMs) can generate efficient programs. However, results on these benchmarks often show that LLM-generated implementations have little or no execution-time difference from canonical solutions. In this paper, we revisit four popular benchmarks: EffiBench, Enamel, EvalPerf, and Mercury. We evaluate 1,538 tasks under more rigorous setting by running each task 30 times and assessing the runtime differences between the canonical solutions and benchmark-provided performant implementations with statistical testing. With the benchmark-provided test suites, only 6.11% of the performant implementations are significantly faster than the canonical solutions. In a manual analysis of 308 non-significant tasks, 99 performant implementations contain no meaningful performance change, while 209 contain potential performance improvements that are not exposed by the original tests. These results suggest that the main limitation is not only the evaluation method, but also the limited sufficiency of the benchmark-provided performance tests. To address this limitation, we propose an LLM-based multi-agent framework to generate performance-oriented tests that expose runtime differences more effectively than the original tests. The framework uses three separate agents to generate, diagnose, and repair deterministic tests that preserve functional correctness while better exposing performance differences. Across 1,345 benchmark tasks for which the original tests found no significant performance difference, tests generated by our framework with DeepSeek-v3.1 and GPT-4o reveal statistically significant improvements in 24.01% and 25.43% of the tasks, respectively, outperforming the SOTA LLM-based performance test generation method.

Related papers

• PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization [18.924749191192433]
  Performance optimization is a distinct agentic task.<n>We introduce PERFOPT-Bench, a benchmark for evaluating this full performance-engineering loop.
  arXiv  Detail & Related papers  (2026-07-08T08:32:18Z)
• Evaluating LLMs on Real-World Software Performance Optimization [11.535003393431348]
  We introduce SWE-Pro, a repository-level benchmark derived from 102 expert-written optimizations from open-source projects.<n>Unlike previous benchmarks, SWE-Pro pairs each task with parameterized tests to evaluate runtime, peak memory, and Time-Weighted Memory Usage.<n>Our evaluation shows that current Large Language Models struggle significantly: runtime gains are negligible, and memory optimizations are nearly non-existent.
  arXiv  Detail & Related papers  (2026-06-24T08:07:41Z)
• DEP: A Decentralized Large Language Model Evaluation Protocol [51.3646001384887]
  Decentralized Evaluation Protocol (DEP) is a decentralized yet unified and standardized evaluation framework.<n>By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation.<n>We develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control.
  arXiv  Detail & Related papers  (2026-03-01T16:10:16Z)
• Structured Prompting Enables More Robust Evaluation of Language Models [38.53918044830268]
  We present a DSPy+HELM framework that introduces structured prompting methods which elicit reasoning.<n>We find that without structured prompting, HELM underestimates LM performance (by 4% average) and performance estimates vary more across benchmarks.<n>This is the first benchmarking study to systematically integrate structured prompting into an established evaluation framework.
  arXiv  Detail & Related papers  (2025-11-25T20:37:59Z)
• Fluid Language Model Benchmarking [126.92394365620525]
  We introduce Fluid Benchmarking, a new evaluation approach that advances LM benchmarking across multiple dimensions.<n>Inspired by psychometrics, Fluid Benchmarking is based on the insight that the relative value of benchmark items depends on an LM's capability level.<n>We examine four dimensions -- efficiency, validity, variance, and saturation -- and find that Fluid Benchmarking achieves superior performance in all of them.
  arXiv  Detail & Related papers  (2025-09-14T05:49:42Z)
• Benchmarking LLMs for Unit Test Generation from Real-World Functions [34.70460519870186]
  We introduce ULT (UnLeakedTestbench), a new benchmark specifically designed for function-level unit test generation from real-world Python functions.<n>With 3,909 carefully selected function-level tasks, ULT provides a more realistic and challenging evaluation of LLMs' test generation capabilities.<n>Our evaluation results demonstrate that ULT is significantly more challenging.
  arXiv  Detail & Related papers  (2025-08-01T08:08:26Z)
• carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks [61.79411281702448]
  carps is a benchmark framework for Comprehensive Automated Research Performance Studies.<n>We focus on the four most important types of HPO task types: blackbox, multi-objective, multi-fidelity-multi-objective.<n>With 3 336 tasks from 5 community benchmark collections and 28 variants of 9 families, we offer the biggest go-to library to date.
  arXiv  Detail & Related papers  (2025-06-06T15:01:39Z)
• When Should I Run My Application Benchmark?: Studying Cloud Performance Variability for the Case of Stream Processing Applications [1.3398445165628463]
  This paper empirically quantify the impact of cloud performance variability on benchmarking results.<n>With approximately 591 hours of experiments, deploying 789 clusters on AWS and executing 2366 benchmarks, this is likely the largest study of its kind.
  arXiv  Detail & Related papers  (2025-04-16T07:22:44Z)
• CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification [71.34070740261072]
  This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases.<n>The benchmark is containerized for code execution across tasks, and we will release the code, data, and construction methodologies.
  arXiv  Detail & Related papers  (2025-02-12T21:42:56Z)
• Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation [61.350306618479365]
  Leakage of benchmarks can prevent the accurate assessment of large language models' true performance. We propose Inference-Time Decontamination (ITD) to address this issue. ITD reduces inflated accuracy by 22.9% on GSM8K and 19.0% on MMLU.
  arXiv  Detail & Related papers  (2024-06-20T04:35:59Z)
• The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance [0.0]
  We show that the base-rate probability (BRP) differences across answer tokens are significant and affect task performance ie. guess A if uncertain. We propose the Nvr-X-MMLU task, a variation of MMLU, which helps to disambiguate test-taking ability from task performance and reports the latter.
  arXiv  Detail & Related papers  (2024-06-17T15:14:10Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.