SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

• URL: http://arxiv.org/abs/2603.24755v1
• Date: Wed, 25 Mar 2026 19:26:44 GMT
• Title: SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
• Abstract summary: We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
• Score: 55.76734816061826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software development is iterative, yet agentic coding benchmarks overwhelmingly evaluate single-shot solutions against complete specifications. Code can pass the test suite but become progressively harder to extend. Recent iterative benchmarks attempt to close this gap, but constrain the agent's design decisions too tightly to faithfully measure how code quality shapes future extensions. We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints, in which agents repeatedly extend their own prior solutions under evolving specifications that force architectural decisions without prescribing internal structure. We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion, the share of complexity mass concentrated in high-complexity functions. No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%. Quality degrades steadily: erosion rises in 80% of trajectories and verbosity in 89.8%. Against 48 open-source Python repositories, agent code is 2.2x more verbose and markedly more eroded. Tracking 20 of those repositories over time shows that human code stays flat, while agent code deteriorates with each iteration. A prompt-intervention study shows that initial quality can be improved, but it does not halt degradation. These results demonstrate that pass-rate benchmarks systematically undermeasure extension robustness, and that current agents lack the design discipline iterative software development demands.

Related papers

• SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring [31.684059339633013]
  SWE-Bench ProMax is an expert-curated, multilingual code benchmark of 170 instances drawn from real commits.<n>Every instance undergoes rigorous, multi-stage curation that addresses quality problems identified in prior benchmarks.<n>Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate.
  arXiv  Detail & Related papers  (2026-08-10T16:23:19Z)
• How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions [16.8655550230265]
  We operationalize misalignment as a breakdown made visible through developer pushback.<n>We identify seven recurring forms, spanning how agents read projects, interpret developer intent, follow rules, bound their actions, implement and execute code, and report progress.<n>Our findings inform the design of training, evaluation, and interfaces for keeping coding agents aligned with real developer.
  arXiv  Detail & Related papers  (2026-05-28T06:35:39Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development [49.63491095660809]
  ProjDevBench is an end-to-end benchmark that provides project requirements to coding agents and evaluates the resulting repositories.<n>We curate 20 programming problems across 8 categories, covering both concept-oriented tasks and real-world application scenarios.<n>Our evaluation reports an overall acceptance rate of 27.38%: agents handle basic functionality but struggle with complex system design, time optimization, and resource management.
  arXiv  Detail & Related papers  (2026-02-02T05:17:23Z)
• SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios [6.776894728701934]
  Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or implementing a small feature.<n>We introduce SWE-EVO, a benchmark that evaluates agents on a long-horizon software evolution challenge.<n>Tool comprises 48 evolution tasks that require agents to implement multi-step modifications spanning an average of 21 files.
  arXiv  Detail & Related papers  (2025-12-20T19:08:15Z)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents [79.29376673236142]
  Existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems.<n>We present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents.
  arXiv  Detail & Related papers  (2025-12-14T15:12:13Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios [13.949319911378826]
  This study evaluated 4,892 patches from 10 top-ranked agents on 500 real-world GitHub issues.<n>No single agent dominated, with 170 issues unresolved, indicating room for improvement.<n>Most agents maintained code reliability and security, avoiding new bugs or vulnerabilities.<n>Some agents increased code complexity, many reduced code duplication and minimized code smells.
  arXiv  Detail & Related papers  (2024-10-16T11:33:57Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.