SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks

• URL: http://arxiv.org/abs/2606.05574v1
• Date: Thu, 04 Jun 2026 01:49:07 GMT
• Title: SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks
• Abstract summary: SmellBench proactively injects code smells into clean code from real-world repositories.<n>It contains 294 cases spanning 7 popular smell types, 3 difficulty levels, 2 instruction settings across 7 real-world repositories.<n>Experiments show that the best combination - Qwenwen Code + Sonnet 4.5 - achieved only a 50.34 score of smell elimination.
• Score: 34.43588103259487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code Agents have achieved remarkable advances in recent years, exhibiting strong capabilities across a wide range of software engineering tasks. However, their misuse often produces bloated and disorganized code that impairing readability, extensibility, and robustness. Despite this risk, existing benchmarks largely evaluate functional correctness rather than long-term maintainability of code agents. In this paper, we propose SmellBench, an extensible code refactoring benchmark that proactively injects code smells into clean code snippets from real-world repositories. This design enables the generation of controlled, high-quality, and diverse refactoring cases with human-written ground truth. Specifically, it contains 294 cases spanning 7 popular smell types, 3 difficulty levels, 2 instruction settings across 7 real-world repositories. We further design 3 evaluation aspects covering functional correctness, localization ability, and refactoring quality assessment. Experiments with 2 popular agents and 6 large langauge models (LLMs) show that the best combination - Qwen Code + Claude Sonnet 4.5 - achieved only a 50.34 score of smell elimination. Further analysis reveals that this gap arises from a focus on local code smells and a lack of cross-file understanding, which hinders comprehensive smell elimination.

Related papers

• SmellCC: A Tool for Automated Code Smells Remediation [31.421826190169284]
  Code smells significantly threaten software maintainability by accumulating technical debt.<n>We present SmellCC, a Visual Studio Code extension that augments SonarQube with an LLM-based pipeline to automatically detect and remediate Python code smells.<n>SmellCC provides in-place, one-click remediation for the top-10 most frequent smells, effectively preventing the accumulation of technical debt during development.
  arXiv  Detail & Related papers  (2026-08-10T11:45:23Z)
• Is Agent Code Less Maintainable Than Human Code? [17.226150835020146]
  We investigate how agent code compares to human code in maintenance settings.<n>We find that agents are less effective at resolving tasks when building on agent code compared to human code.
  arXiv  Detail & Related papers  (2026-06-19T23:56:14Z)
• Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study [0.18907108368038217]
  We show that code cleanliness affects an agent's ability to navigate and modify it.<n>Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development.
  arXiv  Detail & Related papers  (2026-05-19T16:06:26Z)
• SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair [1.6922906233636834]
  Architectural code smells erode software maintainability and are costly to repair manually.<n>We present the first empirical evaluation of large language model agents on architectural code smell repair.<n>We contribute SmellBench, a task orchestration framework that incorporates smell-type-specific optimized prompts.
  arXiv  Detail & Related papers  (2026-05-07T22:33:32Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• Readability-Robust Code Summarization via Meta Curriculum Learning [53.44612630063336]
  In the real world, code is often poorly structured or obfuscated, significantly degrading model performance.<n>We propose RoFTCodeSum, a novel fine-tuning method that enhances the robustness of code summarization against poorly readable code.
  arXiv  Detail & Related papers  (2026-01-09T02:38:24Z)
• A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code [49.09545217453401]
  Propensity Smelly Score (PSC) is a metric that estimates the likelihood of generating particular smell types.<n>We identify how generation strategy, model size, model architecture and prompt formulation shape the structural properties of generated code.<n> PSC helps developers interpret model behavior and assess code quality, providing evidence that smell propensity signals can support human judgement.
  arXiv  Detail & Related papers  (2025-11-19T19:18:28Z)
• Clean Code, Better Models: Enhancing LLM Performance with Smell-Cleaned Dataset [13.23492570818459]
  This study takes the first systematic research to assess and improve dataset quality in terms of code smells.<n>We propose an LLM-based code smell cleaning tool, named SmellCC, which automatically removes code smells.
  arXiv  Detail & Related papers  (2025-08-16T07:40:58Z)
• Turning the Tide: Repository-based Code Reflection [52.13709676656648]
  We introduce LiveRepoReflection, a benchmark for evaluating code understanding and generation in multi-file repository contexts.<n>1,888 rigorously filtered test cases across $6$ programming languages to ensure diversity, correctness, and high difficulty.<n>We also create RepoReflection-Instruct, a large-scale, quality-filtered instruction-tuning dataset derived from diverse sources.
  arXiv  Detail & Related papers  (2025-07-14T02:36:27Z)
• Understanding Code Understandability Improvements in Code Reviews [79.16476505761582]
  We analyzed 2,401 code review comments from Java open-source projects on GitHub. 83.9% of suggestions for improvement were accepted and integrated, with fewer than 1% later reverted.
  arXiv  Detail & Related papers  (2024-10-29T12:21:23Z)
• Empirical Analysis on Effectiveness of NLP Methods for Predicting Code Smell [3.2973778921083357]
  A code smell is a surface indicator of an inherent problem in the system. We use three Extreme learning machine kernels over 629 packages to identify eight code smells. Our findings indicate that the radial basis functional kernel performs best out of the three kernel methods with a mean accuracy of 98.52.
  arXiv  Detail & Related papers  (2021-08-08T12:10:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.