Related papers: SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

URL: http://arxiv.org/abs/2608.09802v1
Date: Mon, 10 Aug 2026 16:23:19 GMT
Title: SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Abstract summary: SWE-Bench ProMax is an expert-curated, multilingual code benchmark of 170 instances drawn from real commits.<n>Every instance undergoes rigorous, multi-stage curation that addresses quality problems identified in prior benchmarks.<n>Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate.
Score: 31.684059339633013
License: http://creativecommons.org/licenses/by/4.0/
Abstract: As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.

Related papers

SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents [63.65682461568621]
We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness.<n>SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints.<n>Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance.
arXiv Detail & Related papers (2026-09-03T17:53:34Z)
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks [1.7623000586936592]
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.<n>Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream.<n>Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code.
arXiv Detail & Related papers (2026-07-08T21:45:34Z)
Automating Formal Verification with Agent-Guided Tree Search [0.0]
Formal verification offers a path to provably correct software, but writing verified code remains expensive enough that the technique is rarely used in production.<n>Recent benchmarks measure their ability to translate specifications into code and machine-checked iteration of correctness.<n>This thesis evaluates the state of such LLM-driven verified-code generation in Lean and develops search-based methods for improving performance.
arXiv Detail & Related papers (2026-05-26T14:50:47Z)
LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
arXiv Detail & Related papers (2026-05-02T11:31:33Z)
Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
arXiv Detail & Related papers (2026-04-02T01:13:40Z)
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
arXiv Detail & Related papers (2026-03-25T19:26:44Z)
SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models [59.90381306452982]
evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer.<n>We introduce SWE-1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework.<n>SWE- spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests.
arXiv Detail & Related papers (2025-11-07T18:01:32Z)
ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
"Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
arXiv Detail & Related papers (2025-10-23T06:58:32Z)
UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench [8.00058513405915]
We introduce UTGenerator, an LLM-driven test case generator to generate test cases for real-world Python projects.<n>Building on UTGenerator, we propose UTBoost, a comprehensive framework for test case augmentation.<n>In our evaluation, we identified 36 task instances with insufficient test cases and uncovered 345 erroneous patches incorrectly labeled as passed in the original SWE Bench.
arXiv Detail & Related papers (2025-06-10T22:56:49Z)
SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents [10.730852617039451]
We investigate the capability of LLM-based Code Agents to formalize user issues into test cases.<n>We propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests.<n>We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed for test generation.
arXiv Detail & Related papers (2024-06-18T14:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.