SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- URL: http://arxiv.org/abs/2605.08366v1
- Date: Fri, 08 May 2026 18:21:44 GMT
- Title: SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- Abstract summary: SWE Atlas is a benchmark suite for coding agents spanning three professional software engineering: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks)<n>It targets underrepresented but practically important categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified rubrics.<n>Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.
- Score: 16.25554650122462
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.
Related papers
- Benchmarking the Titans: A Multi-Dimensional Empirical Evaluation of LLM Code Generation Quality in the .NET Ecosystem [41.99844472131922]
Existing benchmarks reduce evaluation to a single Pass@k metric, which obscures critical trade-offs between functional correctness and structural quality.<n>This paper presents an automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art Large Language Model (LLM)<n>We conduct a controlled experiment across 85 algorithmic tasks derived from HumanEval, generating and evaluating 340 solutions in total, in which each solution is assessed across three independent dimensions.
arXiv Detail & Related papers (2026-08-23T17:59:19Z) - AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs [55.8213573429699]
AISE-Bench is a real-world, full-cycle annotated benchmark for information seeking on academic knowledge graphs.<n>AISE-Bench release contains 1,133 QA pairs, including query, full API execution, validated parameters, and source-grounded answers with reference links.<n>We develop a comprehensive evaluation protocol measuring answer quality, reference grounding, API-planning correctness, and execution success.
arXiv Detail & Related papers (2026-06-16T07:59:46Z) - ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development [49.63491095660809]
ProjDevBench is an end-to-end benchmark that provides project requirements to coding agents and evaluates the resulting repositories.<n>We curate 20 programming problems across 8 categories, covering both concept-oriented tasks and real-world application scenarios.<n>Our evaluation reports an overall acceptance rate of 27.38%: agents handle basic functionality but struggle with complex system design, time optimization, and resource management.
arXiv Detail & Related papers (2026-02-02T05:17:23Z) - SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models [59.90381306452982]
evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer.<n>We introduce SWE-1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework.<n>SWE- spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests.
arXiv Detail & Related papers (2025-11-07T18:01:32Z) - When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs [0.0]
This paper presents a streamlined evaluation framework using twenty carefully designed prompts to assess instruction-following.<n>We demonstrate this framework through a large-scale empirical study conducted on October 14, 2025.<n>Our findings reveal consistent failure modes and identify specific instruction types posing particular challenges.
arXiv Detail & Related papers (2025-10-18T16:33:15Z) - Establishing Best Practices for Building Rigorous Agentic Benchmarks [94.69724201080155]
We show that many agentic benchmarks have issues in task setup or reward design.<n>Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms.<n>We introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience.
arXiv Detail & Related papers (2025-07-03T17:35:31Z) - carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks [61.79411281702448]
carps is a benchmark framework for Comprehensive Automated Research Performance Studies.<n>We focus on the four most important types of HPO task types: blackbox, multi-objective, multi-fidelity-multi-objective.<n>With 3 336 tasks from 5 community benchmark collections and 28 variants of 9 families, we offer the biggest go-to library to date.
arXiv Detail & Related papers (2025-06-06T15:01:39Z) - Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks [16.696044684014435]
Large language models (LLMs) are gaining increasing popularity in software engineering (SE)<n> evaluating their effectiveness is crucial for understanding their potential in this field.<n>This paper offers a thorough review of 291 benchmarks, addressing three main aspects: what benchmarks are available, how benchmarks are constructed, and the future outlook for these benchmarks.
arXiv Detail & Related papers (2025-05-13T18:45:10Z) - HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems [2.4241401076864]
The HackerRank-ASTRA Benchmark introduces project-based coding problems that mirror real-world scenarios.<n>It evaluates model consistency through 32 runs (k = 32) and median standard deviation.<n>The top three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved comparable average scores of 75%.
arXiv Detail & Related papers (2025-01-31T23:47:02Z) - The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
arXiv Detail & Related papers (2024-12-06T23:43:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.