MirrorCode: AI can rebuild entire programs from behavior alone

• URL: http://arxiv.org/abs/2606.30182v1
• Date: Mon, 29 Jun 2026 11:57:32 GMT
• Title: MirrorCode: AI can rebuild entire programs from behavior alone
• Abstract summary: MirrorCode is a long-horizon coding benchmark based on reimplementing entire software projects.<n>Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark.<n>We show that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified.
• Score: 1.052679131856939
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI models are rapidly improving at autonomous coding, as shown by benchmark progress and one-off demonstrations such as AI implementing a C compiler. However, existing coding benchmarks tend to focus on shorter tasks, and one-off demonstrations are hard to compare systematically because they often have some human guidance, and are not standardized or repeated across models. To address these challenges, we introduce MirrorCode, a long-horizon coding benchmark based on reimplementing entire software projects. In MirrorCode, AI agents must replicate the functionalities of an existing program, without access to its source code. AI solutions must match the original program's output exactly on end-to-end tests, including held-out tests. MirrorCode's 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark. For example, AI can reimplement gotree, a 16,000-line bioinformatics toolkit - a task that we believe would take weeks for a human engineer. However, studying the frontier of performance requires a larger inference budget than typical benchmarks, for example, \$2,600 over 19 days for a single attempt on a large task. We show that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified. More broadly, our work suggests AI will have transformative effects on software engineering, as autonomous agents continue to improve.

Related papers

• Accelerated Genetic Programming Hyper-Heuristics for Simulation-Based Scheduling via Agentic AI [42.44812101223683]
  This paper presents a systematic approach using Claude agentic AI on real-world project-scheduling workloads in a high-performance computing environment.<n>Testing runtime reduced from 1,298 seconds to under 200 seconds without changing outputs, saving four million core-hours (NZ$320,000) annually.
  arXiv  Detail & Related papers  (2026-08-19T22:48:46Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• ProgramBench: Can Language Models Rebuild Programs From Scratch? [59.40748183470308]
  ProgramBench measures the ability of software engineering agents to develop software holisitically.<n>End-to-end behavioral tests are generated via agent-driven fuzzing.<n>Models favor monolithic, single-file implementations that diverge sharply from human-written code.
  arXiv  Detail & Related papers  (2026-05-05T09:17:02Z)
• LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces [65.11019654023978]
  LongCLI-Bench is a benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks.<n>We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world tasks.<n>Experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench.
  arXiv  Detail & Related papers  (2026-02-15T23:12:57Z)
• SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents [21.8776989802963]
  SWE-AGI is an open-source benchmark for evaluating end-to-end, specification-driven construction of software systems written in MoonBit.<n>Each task involves implementing 1,000-10,000 lines of core logic, corresponding to weeks or months of engineering effort for an experienced human developer.
  arXiv  Detail & Related papers  (2026-02-10T06:31:47Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• GAICo: A Deployed and Extensible Framework for Evaluating Diverse and Multimodal Generative AI Outputs [8.34331981959369]
  We present GAICo (Generative AI Comparator): a deployed, open-source Python library that standardizes GenAI output comparison.<n> GAICo provides a unified, framework supporting a comprehensive suite of reference-based metrics for unstructured text, structured data formats, and multimedia.<n>Since its release on PyPI in Jun 2025, the tool has been downloaded over 13K times, across versions, by Aug 2025, demonstrating growing community interest.
  arXiv  Detail & Related papers  (2025-08-22T19:13:21Z)
• An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We? [8.0988059417354]
  We propose a range of approaches to improve the performance of AI-generated code detection. Our best model outperforms state-of-the-art AI-generated code detector (GPTSniffer) and achieves an F1 score of 82.55.
  arXiv  Detail & Related papers  (2024-11-06T22:48:18Z)
• AutoCodeRover: Autonomous Program Improvement [8.66280420062806]
  We propose an automated approach for solving GitHub issues to autonomously achieve program improvement. In our approach called AutoCodeRover, LLMs are combined with sophisticated code search capabilities, ultimately leading to a program modification or patch. Experiments on SWE-bench-lite (300 real-life GitHub issues) show increased efficacy in solving GitHub issues (19% on SWE-bench-lite), which is higher than the efficacy of the recently reported SWE-agent.
  arXiv  Detail & Related papers  (2024-04-08T11:55:09Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)
• Measuring Coding Challenge Competence With APPS [54.22600767666257]
  We introduce APPS, a benchmark for code generation. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. Recent models such as GPT-Neo can pass approximately 15% of the test cases of introductory problems.
  arXiv  Detail & Related papers  (2021-05-20T17:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.