EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions

• URL: http://arxiv.org/abs/2605.24110v1
• Date: Fri, 22 May 2026 18:17:28 GMT
• Title: EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
• Abstract summary: EvoCode-Bench is a benchmark of 26 stateful coding tasks and 227 evaluated rounds.<n>Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior.<n>We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state.
• Score: 9.297494684604763
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This leaves out a basic question: can an agent keep its own codebase working as requirements change? We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior, and uses cumulative executable tests to check new requirements and still-active prior ones. We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state. For most agents, SR exceeds MT@4 by 22-40 points. The gap also changes rankings: the highest-SR agent (78.9) ranks only third in persistent execution (44.0 MT@4). Even the strongest agents achieve only about 50% success on multi-turn metrics, and aggregate pass rate drops below half of round-1 performance by round 5. Failure analysis shows tier-dependent behavior: weaker agents fail early, while stronger agents survive long enough to expose specification-tracking and regression failures. We release the benchmark data and Harbor multi-turn infrastructure.

Related papers

• Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course [5.9756241643216255]
  Large language model (LLM)-powered agents can be accurate on average yet unreliable in production.<n>We present a self-evolving framework that reduces this gap by identifying unstable, low-consistency steps in agent trajectories.<n>On AppWorld with ReAct/GPT-4.1, our framework raises the fraction of tasks that succeed in all five runs by +16 points on same-task evaluation and +13 points on similar-task generalization.
  arXiv  Detail & Related papers  (2026-09-08T14:53:43Z)
• Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair [13.668549452625117]
  Kozuchi Agent is a language-agnostic open-weight repair agent and CI-operated evaluation pipeline.<n>Explicit phases, persistent state, deterministic tools, a model-independent action interface, and cross-agent test-time selection make runs auditable and repeatable.
  arXiv  Detail & Related papers  (2026-08-16T07:06:37Z)
• Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds [43.5891705224868]
  Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model.<n>We present a controlled evaluation framework across five benchmarks and three models.<n>Overall, persistent skill self-evolution is better understood as sparse, validation-filtered search with model- and benchmark-dependent returns.
  arXiv  Detail & Related papers  (2026-07-31T04:02:51Z)
• MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolution [24.78476085778134]
  Self-improving agents address this by rewriting their own skill files from execution traces, yielding meaningful gains on challenging benchmarks.<n>We introduce MetaSkill-Evolve, a two-timescale framework that makes agentic skill improvement recursive.<n>Task skills evolve on a fast loop while the meta-skill evolves on a slower one under the same pipeline applied to itself.<n>With all five pipeline agents sharing a single frozen backbone, MetaSkill-Evolve outperforms no-skill, static-skill, and single-level evolution baselines.
  arXiv  Detail & Related papers  (2026-07-06T16:40:23Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• SNARE: Adaptive Scenario Synthesis for Eliciting Overeager Behavior in Coding Agents [40.270213696031625]
  A coding agent executes a benign task as a sequence of shell, file, and network actions, any of which can quietly exceed the authorized scope while the task still completes.<n>We call this overeager behavior: the prompt is not adversarial and the run succeeds, yet an out-of-scope step can leak credentials or delete files.<n>Existing benchmarks miss it: task-completion suites credit any finished run, jailbreak suites probe adversarial prompts, and the one prior overeager benchmark applies a single fixed prompt to every agent-model pair.<n>We present SNARE, a pipeline that composes benign scenarios from reusable scope
  arXiv  Detail & Related papers  (2026-05-27T08:14:07Z)
• AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation [11.272830796781925]
  We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks.<n>Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass.<n>We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories.
  arXiv  Detail & Related papers  (2026-05-13T03:00:57Z)
• Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values [17.752804388034583]
  Agent-ValueBench is the first benchmark dedicated to agent values.<n>It features 394 executable environments across 16 domains, offering 4,335 value-conflict tasks.<n>Agent values first manifest as a Value Tide of cross-model homogeneity beneath interpretable counter-currents.
  arXiv  Detail & Related papers  (2026-05-11T11:09:04Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.