SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements

• URL: http://arxiv.org/abs/2607.27409v1
• Date: Wed, 29 Jul 2026 19:28:32 GMT
• Title: SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
• Abstract summary: We present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness.<n>We develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation.
• Score: 25.010113476664923
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents' NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.

Related papers

• REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming [0.0]
  Large language models (LLMs) are increasingly applied to reverse-engineering tasks.<n>Claims about their capability, however, outpace our ability to measure it.<n>This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source.
  arXiv  Detail & Related papers  (2026-07-07T23:17:03Z)
• The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators [17.44053709945519]
  Self-improving agents are state-of-the-art (SOTA) on agentic coding benchmarks and have recently been extended to general domains.<n>We introduce the Red Queen Godel Machine (RQGM), an evolutionary framework for self-improvement under non-stationary utilities.<n>We show that the RQGM improves test pass rate over the prior SOTA by adding a complementary agent-as-a-judge code-review signal.<n>We then turn to scientific paper writing and reviewing, and Olympiad-level proof writing and grading, where the RQGM improves performance over prior self-improving agents
  arXiv  Detail & Related papers  (2026-06-24T18:38:26Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation [59.964989458924585]
  AdaExplore is an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation.<n>In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules.<n>In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration.
  arXiv  Detail & Related papers  (2026-04-17T18:25:03Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• FeatureBench: Benchmarking Agentic Coding for Complex Feature Development [42.26354337364403]
  FeatureBench is a benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development.<n>It incorporates an execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort.<n> Empirical evaluation reveals that the state-of-the-art agentic model, such as Claude 4.5 Opus, achieves a 74.4% resolved rate on SWE-bench.
  arXiv  Detail & Related papers  (2026-02-11T16:06:32Z)
• InnoGym: Benchmarking the Innovation Potential of AI Agents [74.64144272881414]
  InnoGym is the first benchmark designed to evaluate the innovation potential of AI agents.<n>InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches.
  arXiv  Detail & Related papers  (2025-12-01T16:03:04Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• PerfBench: Can Agents Resolve Real-World Performance Bugs? [4.879400115033142]
  PerfBench is a benchmark comprising 81 real-world performance bug-fixing tasks from GitHub.<n>PerfBench features a novel evaluation harness that allows agents to generate their own performance benchmarks.<n>We develop OpenHands-Perf-Agent, which incorporates performance-aware tooling and instructions and achieves a 20% success rate on the benchmark.
  arXiv  Detail & Related papers  (2025-09-28T22:00:33Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generation [4.065344017083881]
  We analyze the ability of embedding-based metrics like CodeBERTScore to measure functional correctness and other helpful constructs like editing effort. Our results show that while they have a weak correlation with functional correctness (0.16), they are strongly correlated (0.72) with editing effort.
  arXiv  Detail & Related papers  (2024-04-26T15:54:39Z)
• NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness [10.502272765892908]
  Existing evaluation benchmarks of language models of code (code LMs) focus almost exclusively on whether the LMs can generate functionally-correct code. We propose a new benchmark NoFunEval to evaluate code LMs on non-functional requirements and simple classification instances for both functional and non-functional requirements. Our finding is that LMs generally falter when tested on our benchmark, hinting at fundamental blindspots in their training setups.
  arXiv  Detail & Related papers  (2024-01-29T08:47:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.