RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades

• URL: http://arxiv.org/abs/2605.15846v2
• Date: Tue, 19 May 2026 08:10:44 GMT
• Title: RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
• Abstract summary: We present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades.<n>Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction.<n>We find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest iteration achieves merely 5.2%.
• Score: 32.04167306620878
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem.

Related papers

• LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent [60.27239351179265]
  LOOPSBENCH is a benchmark for loop engineering in coding agent evaluation.<n>It comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains.<n>Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations.
  arXiv  Detail & Related papers  (2026-07-31T20:18:25Z)
• MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis [17.260324148993046]
  We introduce MindForge, an automated pipeline that converts open-source command-line programs into source-free environments.<n>We construct training environments from repositories disjoint from those in ProgramBench, and curate a high-quality data recipe.<n>Fine-tuning Qwen3.6-27B on these trajectories increases its ProgramBench average test pass rate from 37.98% to 49.51%.
  arXiv  Detail & Related papers  (2026-07-29T17:23:02Z)
• ProgramBench: Can Language Models Rebuild Programs From Scratch? [59.40748183470308]
  ProgramBench measures the ability of software engineering agents to develop software holisitically.<n>End-to-end behavioral tests are generated via agent-driven fuzzing.<n>Models favor monolithic, single-file implementations that diverge sharply from human-written code.
  arXiv  Detail & Related papers  (2026-05-05T09:17:02Z)
• MobileDev-Bench: A Comprehensive Benchmark for Evaluating Language Models on Mobile Application Development [1.4302803674538154]
  We introduce MobileDev-Bench, a benchmark comprising 384 real-world issue-resolution tasks collected from 18 production mobile applications spanning Android Native (Java/Kotlin), React Native (TypeScript), and Flutter (Dart)<n>Each task pairs an authentic developer-reported issue with executable test patches, enabling fully automated validation of model-generated fixes within mobile build environments.<n> Evaluation of four state-of-the-art code-capable LLMs, GPT- 5.2, Claude Sonnet 4.5, Gemini Flash 2.5, and Qwen3-Coder, yields low end-to-end resolution rates of 3.39%
  arXiv  Detail & Related papers  (2026-03-26T02:31:03Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? [61.247730037229815]
  We introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope.<n>To investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities.<n>This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
  arXiv  Detail & Related papers  (2026-03-03T17:52:01Z)
• SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios [6.776894728701934]
  Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or implementing a small feature.<n>We introduce SWE-EVO, a benchmark that evaluates agents on a long-horizon software evolution challenge.<n>Tool comprises 48 evolution tasks that require agents to implement multi-step modifications spanning an average of 21 files.
  arXiv  Detail & Related papers  (2025-12-20T19:08:15Z)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents [79.29376673236142]
  Existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems.<n>We present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents.
  arXiv  Detail & Related papers  (2025-12-14T15:12:13Z)
• The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution [86.4588675093384]
  Toolathlon is a benchmark for language agents offering diverse Apps and tools, realistic environment setup, and reliable execution-based evaluation.<n>This benchmark includes 108 manually sourced or crafted tasks, requiring interacting with multiple Apps over around 20 turns on average to complete.<n>We expect Toolathlon to drive the development of more capable language agents for real-world, long-horizon task execution.
  arXiv  Detail & Related papers  (2025-10-29T17:32:49Z)
• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? [13.645265361867565]
  SWE-Bench Pro builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems.<n>The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories.<n>In our evaluation of widely used coding models, under a unified scaffold, we observe that their performance on SWE-Bench PRO remains below 25% (Pass@1), with GPT-5 achieving the highest score to date at 23.3%.
  arXiv  Detail & Related papers  (2025-09-21T06:28:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.