Orchestrating Human-AI Software Delivery: A Retrospective Longitudinal Field Study of Three Software Modernization Programs

• URL: http://arxiv.org/abs/2603.20028v1
• Date: Fri, 20 Mar 2026 15:14:36 GMT
• Title: Orchestrating Human-AI Software Delivery: A Retrospective Longitudinal Field Study of Three Software Modernization Programs
• Abstract summary: We report a longitudinal field study of Chiron, an industrial platform that coordinates humans and AI agents across four delivery stages.<n>The study covers three real software modernization programs -- a banking migration (30k LOC), a large accounting modernization (400k LOC) and a.NET/Angular mortgage modernization (30k LOC)<n>The benchmark separates observed outcomes (stage durations, task volumes, validation-stage issues, first-release coverage) from modeled outcomes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Evidence on AI in software engineering still leans heavily toward individual task completion, while evidence on team-level delivery remains scarce. We report a retrospective longitudinal field study of Chiron, an industrial platform that coordinates humans and AI agents across four delivery stages: analysis, planning, implementation, and validation. The study covers three real software modernization programs -- a COBOL banking migration (~30k LOC), a large accounting modernization (~400k LOC), and a .NET/Angular mortgage modernization (~30k LOC) -- observed across five delivery configurations: a traditional baseline and four successive platform versions (V1--V4). The benchmark separates observed outcomes (stage durations, task volumes, validation-stage issues, first-release coverage) from modeled outcomes (person-days and senior-equivalent effort under explicit staffing scenarios). Under baseline staffing assumptions, portfolio totals move from 36.0 to 9.3 summed project-weeks; modeled raw effort falls from 1080.0 to 232.5 person-days; modeled senior-equivalent effort falls from 1080.0 to 139.5 SEE-days; validation-stage issue load falls from 8.03 to 2.09 issues per 100 tasks; and first-release coverage rises from 77.0% to 90.5%. V3 and V4 add acceptance-criteria validation, repository-native review, and hybrid human-agent execution, simultaneously improving speed, coverage, and issue load. The evidence supports a central thesis: the largest gains appear when AI is embedded in an orchestrated workflow rather than deployed as an isolated coding assistant.

Related papers

• ProCUA-SFT Technical Report [80.97543110323542]
  Training computer-use agents (CUAs) requires large-scale, diverse trajectory data collected in full desktop environments.<n>We present ProCUA-SFT, a dataset of 3.1M step-level SFT samples distilled from 93K synthetic trajectories.<n>Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields 45.0% on OSWorld -- an 18.7 percentage-point improvement over the base model.
  arXiv  Detail & Related papers  (2026-06-15T22:04:11Z)
• Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam? [90.39633603556767]
  Large Language Model (LLM) agents for computer automation is accelerating, yet their ability to navigate complex, professional-grade productivity software is largely untested.<n>To quantify this capability, we introduce an evaluation based on China's National Computer Rank Examination (NCRE)<n>We benchmark 7 frontier LLMs and observe stark limitations: single-turn models score a maximum of 36.6%.<n>A stronger agentic system with execution feedback, iterative repair, and broader Office automation access reaches 68.8%, but remains below the 95.5% community-reference score used as a scoring sanity check.
  arXiv  Detail & Related papers  (2026-06-09T14:59:14Z)
• SPOQ: Specialist Orchestrated Queuing for Multi-Agent Software Engineering [2.0769172070951067]
  Multi-agent AI systems show promise for automating software engineering tasks.<n>Existing approaches suffer from coordination overhead, quality control gaps, and limited human oversight.<n>We introduce SPOQ (Specialist Orchestrated Queuing), a methodology combining three innovations.
  arXiv  Detail & Related papers  (2026-06-02T03:59:56Z)
• Episodic-Semantic Memory Architecture for Long-Horizon Scientific Agents [0.0]
  We evaluate a Dual Process Memory Architecture that decouples immediate episodic needs (constant 10-message window) from long-term consolidated knowledge (growing at approximately 3 tokens/message)<n>Our system maintains 70-85% accuracy with 1-2 second latency using fewer tokens (45,434 vs 120,000+ limit)<n>We identify a "Sim-to-Real" gap where synthetic tests maintain constant memory but realistic exhibit linear growth (about 3 tokens/message)
  arXiv  Detail & Related papers  (2026-05-17T19:44:24Z)
• Orchard: An Open-Source Agentic Modeling Framework [124.68499958175111]
  We present Orchard, an open-source framework for scalable agentic modeling.<n>At its core is Orchard Env, a lightweight environment service providing reusable primitives for sandbox lifecycle management.<n>On top of Orchard Env, we build three agentic modeling recipes.
  arXiv  Detail & Related papers  (2026-05-14T16:35:12Z)
• Agentic AI in the Software Development Lifecycle: Architecture, Empirical Evidence, and the Reshaping of Software Engineering [0.0]
  Modern agentic systems operate at the granularity of a repository, a feature, or an algorithm.<n>We propose a six-layer reference architecture for agentic software engineering systems.<n>We argue that the central object of inquiry has shifted from code generation to delegated execution under human supervision.
  arXiv  Detail & Related papers  (2026-04-29T04:06:47Z)
• OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation [57.505743202759646]
  OccuBench is a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains.<n>Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity.
  arXiv  Detail & Related papers  (2026-04-13T00:27:32Z)
• Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development [6.072381417546439]
  Existing benchmarks measure isolated tasks rather than the complete "zero-to-one" process of building a working application from scratch.<n>We introduce Vibe Code Bench, a benchmark of 100 web application specifications with 964 browser-based substeps.<n>Our contributions include (1) a novel benchmark dataset and browser-based evaluation pipeline for end-to-end web application development, (2) a comprehensive evaluation of 16 frontier models with cost, latency, and error analysis, and (3) an evaluator alignment protocol with both cross-model and human annotation results.
  arXiv  Detail & Related papers  (2026-03-04T21:00:33Z)
• CORPGEN: Simulating Corporate Environments with Autonomous Digital Employees in Multi-Horizon Task Environments [1.6153514666902042]
  Real organizational work requires managing many concurrent long-horizon tasks with interleaving, dependencies, and reprioritization.<n>We introduce Multi-Horizon Task Environments (MHTEs): a distinct problem class requiring coherent execution across dozens of interleaved tasks.<n>We identify four failure modes that cause baseline CUAs to degrade from 16.7% to 8.7% completion as load scales 25% to 100%.<n>We present CorpGen, an architecture-agnostic framework addressing these failures via hierarchical planning for multi-horizon goal alignment.
  arXiv  Detail & Related papers  (2026-02-15T16:54:34Z)
• Comprehensive Evaluation of Large Language Models on Software Engineering Tasks: A Multi-Task Benchmark [0.0]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in software engineering.<n>We present a multi-task evaluation of 11 state-of-the-art LLMs across five representative software engineering tasks.
  arXiv  Detail & Related papers  (2026-02-06T03:30:19Z)
• CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability [50.57373283154859]
  We present CVE-Factory, the first multiagent framework to achieve expert-level quality in automatically transforming vulnerability tasks.<n>It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2% verified success.<n>We synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security.
  arXiv  Detail & Related papers  (2026-02-03T02:27:16Z)
• A Pragmatic VLA Foundation Model [66.76609538850478]
  We develop LingBot-VLA with around 20,000 hours of real-world data from 9 popular dual-arm robot configurations.<n>Our model achieves clear superiority over competitors, showcasing its strong performance and broad generalizability.<n>To advance the field of robot learning, we provide open access to the code, base model, and benchmark data.
  arXiv  Detail & Related papers  (2026-01-26T17:08:04Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains [97.5573252172065]
  We train a family of Automatic Reasoning Evaluators (FARE) with a simple iterative rejection-sampling supervised finetuning approach.<n>FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators.<n>As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH.
  arXiv  Detail & Related papers  (2025-10-20T17:52:06Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Intuition to Evidence: Measuring AI's True Impact on Developer Productivity [30.02516976149379]
  We present a comprehensive real-world evaluation of AI-assisted software development tools deployed at enterprise scale.<n>Over one year, 300 engineers across multiple teams integrated an in-house AI platform (DeputyDev) that combines code generation and automated review capabilities.
  arXiv  Detail & Related papers  (2025-09-24T02:34:11Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems [2.4241401076864]
  The HackerRank-ASTRA Benchmark introduces project-based coding problems that mirror real-world scenarios.<n>It evaluates model consistency through 32 runs (k = 32) and median standard deviation.<n>The top three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved comparable average scores of 75%.
  arXiv  Detail & Related papers  (2025-01-31T23:47:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.