SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

• URL: http://arxiv.org/abs/2607.27167v1
• Date: Wed, 29 Jul 2026 17:42:47 GMT
• Title: SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch
• Abstract summary: LLM-based agents excel at software engineering tasks where an existing context provides context, but constructing a program from scratch remains fundamentally harder.<n>Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass.<n>We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis.
• Score: 17.260324148993046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.

Related papers

• Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study [8.409799680956013]
  General-purpose code embeddings power tools for code search, classification, and retrieval.<n>We study an alternative: contrastive pretraining of small encoders with synthetically generated natural-language descriptions emphasizing code functionality and intent.
  arXiv  Detail & Related papers  (2026-09-03T11:41:09Z)
• Agentic Separation Logic Specification Synthesis [4.177196802192322]
  Spec-Agent is an agentic system for synthesizing expressive, well-validated specifications across large C++s.<n>For each function, Spec-Agent uses static analysis and runtime heap tracing to select the appropriate target specification language.<n>We evaluate Spec-Agent on open source C++s comprising millions of lines of code.
  arXiv  Detail & Related papers  (2026-05-26T18:05:42Z)
• From I/O to Code with Discovery Agent [103.88427301265669]
  We propose DIO-Agent, a discovery agent for IO2Code.<n>Our method frames IO2Code as an evolutionary search over discrete program space.<n>Extensive experiments show that DIO-Agent consistently outperforms both traditional program-by-example method and SOTA evolution-agent baselines.
  arXiv  Detail & Related papers  (2026-05-14T18:57:32Z)
• CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation [49.30536937161147]
  We introduce CodeSpecBench, a benchmark for executable behavioral specification generation under an execution-based evaluation protocol.<n>CodeSpecBench supports both function-level and repository-level tasks and encodes specifications as executable Python functions.<n>We observe a sharp performance degradation on repository-level tasks, where the best model attains only a 20.2% pass rate.
  arXiv  Detail & Related papers  (2026-04-14T04:31:45Z)
• Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining [66.89012795621349]
  Large Language Models (LLMs) often struggle with the deep, long-horizon reasoning required for complex software engineering.<n>We propose a novel paradigm: understanding via reconstruction.<n>We introduce a framework that synthesizes latent agentic trajectories using a multi-agent simulation.
  arXiv  Detail & Related papers  (2026-03-11T09:23:20Z)
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms [54.99368693313797]
  Existing benchmarks test only individual languages/tools, so the performance numbers are not directly comparable.<n>We address this gap with AlgoVeri, a benchmark that evaluates vericoding of $77$ classical algorithms in Dafny, Verus, and Lean.
  arXiv  Detail & Related papers  (2026-02-10T06:58:26Z)
• SynthCoder: A Synthetical Strategy to Tune LLMs for Code Completion [7.668823606571788]
  Code completion is a prominent application of Large Language Models (LLMs) in software engineering.<n>This paper proposes SynthCoder, a model that integrates leading industry practices to achieve state-of-the-art on the Fill-in-the-Middle (FIM) code completion task.
  arXiv  Detail & Related papers  (2025-08-21T12:23:49Z)
• Your Coding Intent is Secretly in the Context and You Should Deliberately Infer It Before Completion [18.177415287934487]
  Large Language Models (LLMs) are increasingly used for function completion in repository-scales.<n>In real-world repositories, such annotations are frequently absent, and performance drops substantially without them.<n>We frame the task as a three-stage process. The first stage focuses on intent inference, where the model analyzes the code preceding the target function to uncover cues about the desired functionality.<n>The second stage introduces an optional interactive refinement mechanism to handle cases where preceding context alone is insufficient for intent recovery.
  arXiv  Detail & Related papers  (2025-08-13T06:45:23Z)
• CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis [6.8081984950459]
  Existing evaluation protocols rely on static sets of examples and held-out tests, offering no feedback when synthesized functions are incorrect.<n>We propose CodeARC, a new evaluation framework where agents interact with a hidden target function by querying it with new inputs.<n>We construct the first large-scale benchmark for general-purpose inductive program synthesis, featuring 1114 functions.
  arXiv  Detail & Related papers  (2025-03-29T16:50:39Z)
• Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond [52.656743602538825]
  Fine-tuning pre-trained code models incurs a large computational cost. We conduct an experimental study to explore what happens to layer-wise pre-trained representations and their encoded code knowledge during fine-tuning. We propose Telly to efficiently fine-tune pre-trained code models via layer freezing.
  arXiv  Detail & Related papers  (2023-04-11T13:34:13Z)
• Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality [84.94877848357896]
  Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. We analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias and the tendency to memorize whole examples. We show substantial empirical improvements using standard sequence-to-sequence models on two widely-used compositionality datasets.
  arXiv  Detail & Related papers  (2022-11-28T17:36:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.