Related papers: ProgramBench: Can Language Models Rebuild Programs From Scratch?

ProgramBench: Can Language Models Rebuild Programs From Scratch?

URL: http://arxiv.org/abs/2605.03546v1
Date: Tue, 05 May 2026 09:17:02 GMT
Title: ProgramBench: Can Language Models Rebuild Programs From Scratch?
Abstract summary: ProgramBench measures the ability of software engineering agents to develop software holisitically.<n>End-to-end behavioral tests are generated via agent-driven fuzzing.<n>Models favor monolithic, single-file implementations that diverge sharply from human-written code.
Score: 59.40748183470308
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95\% of tests on only 3\% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

Related papers

PairCoder++: Pair Programming as a Universal Paradigm for Verified Code-Driven Multimodal and Structured-Artifact Generation [51.92442051257354]
PairCoder improves essentially every benchmark whose artifact is verifiable, on full official metric suites rather than execution alone.<n>TikZ compile rate up 10 to 30 points on every model, at 2.9 to 9.2 times single model cost.
arXiv Detail & Related papers (2026-07-02T08:36:02Z)
MirrorCode: AI can rebuild entire programs from behavior alone [1.052679131856939]
MirrorCode is a long-horizon coding benchmark based on reimplementing entire software projects.<n>Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark.<n>We show that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified.
arXiv Detail & Related papers (2026-06-29T11:57:32Z)
OmniCode: A Benchmark for Evaluating Software Engineering Agents [12.63309178506706]
OmniCode is a novel software engineering benchmark for real-world software development.<n>It contains 1794 tasks spanning three programming languages (Python, Java, and C++) and four key categories: bug fixing, test generation, code review fixing, and style fixing.<n>We evaluate OmniCode with popular agent frameworks such as SWE-Agent and show that while they may perform well on bug fixing for Python, they fall short on tasks such as Test Generation and in languages such as C++ and Java.
arXiv Detail & Related papers (2026-02-02T16:04:10Z)
ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
"Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
arXiv Detail & Related papers (2025-10-23T06:58:32Z)
Programming with Pixels: Can Computer-Use Agents do Software Engineering? [24.011063667060792]
$textttProgramming with Pixels$ (PwP) is the first comprehensive computer-use environment for software engineering.<n>PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance.
arXiv Detail & Related papers (2025-02-24T18:41:33Z)
NExT: Teaching Large Language Models to Reason about Code Execution [50.93581376646064]
Large language models (LLMs) of code are typically trained on the surface textual form of programs. We propose NExT, a method to teach LLMs to inspect the execution traces of programs and reason about their run-time behavior.
arXiv Detail & Related papers (2024-04-23T01:46:32Z)
A Static Evaluation of Code Completion by Large Language Models [65.18008807383816]
Execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models. We propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees.
arXiv Detail & Related papers (2023-06-05T19:23:34Z)
Fault-Aware Neural Code Rankers [64.41888054066861]
We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
arXiv Detail & Related papers (2022-06-04T22:01:05Z)
Measuring Coding Challenge Competence With APPS [54.22600767666257]
We introduce APPS, a benchmark for code generation. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. Recent models such as GPT-Neo can pass approximately 15% of the test cases of introductory problems.
arXiv Detail & Related papers (2021-05-20T17:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.