Evaluating LLMs on Real-World Software Performance Optimization

• URL: http://arxiv.org/abs/2606.25530v1
• Date: Wed, 24 Jun 2026 08:07:41 GMT
• Title: Evaluating LLMs on Real-World Software Performance Optimization
• Abstract summary: We introduce SWE-Pro, a repository-level benchmark derived from 102 expert-written optimizations from open-source projects.<n>Unlike previous benchmarks, SWE-Pro pairs each task with parameterized tests to evaluate runtime, peak memory, and Time-Weighted Memory Usage.<n>Our evaluation shows that current Large Language Models struggle significantly: runtime gains are negligible, and memory optimizations are nearly non-existent.
• Score: 11.535003393431348
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software performance optimization is a notoriously complex and manual task. Despite the growing use of Large Language Models (LLMs) for code refinement, we still lack benchmarks that capture how optimization actually happens in real-world codebases. Existing frameworks often oversimplify the problem by focusing on isolated functions or a single performance metric, missing the critical trade-offs between execution time and memory footprint, the inherent noise of the measurement environment, and the variability introduced by different input data and execution conditions. We address this by introducing SWE-Pro, a repository-level benchmark derived from 102 expert-written optimizations from open-source projects. Unlike previous benchmarks, SWE-Pro pairs each task with parameterized tests to evaluate runtime, peak memory, and Time-Weighted Memory Usage (TWMU) across varying input data and execution conditions under noise-aware measurement conditions. Our evaluation shows that current LLMs struggle significantly: runtime gains are negligible, and memory optimizations are nearly non-existent. This stands in sharp contrast to expert implementations, which achieve an aggregate speedup of 15.5x and peak memory reduction of 171.3x over benchmark tasks. Expert-written improvements are observed in 91.2% of tasks for runtime and 65.7% for peak memory. Our findings expose a substantial gap between current LLM capabilities and the demands of expert-level engineering.

Related papers

• Rethinking Code Performance Benchmarks for LLMs [21.32071427130668]
  We revisit four popular benchmarks: EffiBench, Enamel, EvalPerf, and Mercury.<n>With the benchmark-provided test suites, only 6.11% of the performant implementations are significantly faster than the canonical solutions.<n>We propose an LLM-based multi-agent framework to generate performance-oriented tests that expose runtime differences more effectively than the original tests.
  arXiv  Detail & Related papers  (2026-07-08T16:36:03Z)
• PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization [18.924749191192433]
  Performance optimization is a distinct agentic task.<n>We introduce PERFOPT-Bench, a benchmark for evaluating this full performance-engineering loop.
  arXiv  Detail & Related papers  (2026-07-08T08:32:18Z)
• IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents [56.77101184011525]
  Large language model (LLM)-based agents solve complex tasks by leveraging multi-step reasoning with iterative tool calls and environment interactions.<n>Despite the prevalence of idle time in most agentic scenarios, existing works treat it as an unavoidable overhead.<n>We introduce IdleSpec, a scalable and generic inference approach that leverages idle-time computation to improve agent performance.
  arXiv  Detail & Related papers  (2026-05-21T08:25:17Z)
• The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility [4.514361164656055]
  We show that the choice of backend alone can shift benchmark scores by up to 16.6 percentage points.<n>We find this is driven by system-level optimizations like caching and graphs, custom kernels, and engine-specific defaults in logit processing.
  arXiv  Detail & Related papers  (2026-05-19T08:37:27Z)
• PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization [43.72919370258115]
  We introduce PerfCodeBench, an executable benchmark for evaluating large language models (LLMs) on high-performance code optimization.<n>The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks.<n>Our evaluation shows a clear gap between model-generated code and expert-optimized implementations.
  arXiv  Detail & Related papers  (2026-05-13T08:10:26Z)
• SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? [22.075705411944895]
  SWE-fficiency is a benchmark for evaluating repository-level performance optimization on real workloads.<n>Our suite contains 498 tasks across nine widely used data-science, machine-learning, and HPC repositories.
  arXiv  Detail & Related papers  (2025-11-08T17:55:09Z)
• MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization [103.74675519953898]
  Long-chain reflective reasoning is a prerequisite for solving complex real-world problems.<n>We build a benchmark consisting 1,260 samples of 42 challenging synthetic tasks.<n>We generate post-training data and explore learning paradigms for exploiting such data.
  arXiv  Detail & Related papers  (2025-10-09T17:53:58Z)
• Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation [61.350306618479365]
  Leakage of benchmarks can prevent the accurate assessment of large language models' true performance. We propose Inference-Time Decontamination (ITD) to address this issue. ITD reduces inflated accuracy by 22.9% on GSM8K and 19.0% on MMLU.
  arXiv  Detail & Related papers  (2024-06-20T04:35:59Z)
• Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning [67.44661423463927]
  This paper introduces Sparse MeZO, a memory-efficient zeroth-order optimization approach that applies ZO only to a carefully chosen subset of parameters. We show that Sparse-MeZO consistently improves both performance and convergence speed over MeZO without any overhead.
  arXiv  Detail & Related papers  (2024-02-24T07:22:04Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• Learning Performance-Improving Code Edits [107.21538852090208]
  We introduce a framework for adapting large language models (LLMs) to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs. For prompting, we propose retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play.
  arXiv  Detail & Related papers  (2023-02-15T18:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.