CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation

• URL: http://arxiv.org/abs/2604.12268v1
• Date: Tue, 14 Apr 2026 04:31:45 GMT
• Title: CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
• Abstract summary: We introduce CodeSpecBench, a benchmark for executable behavioral specification generation under an execution-based evaluation protocol.<n>CodeSpecBench supports both function-level and repository-level tasks and encodes specifications as executable Python functions.<n>We observe a sharp performance degradation on repository-level tasks, where the best model attains only a 20.2% pass rate.
• Score: 49.30536937161147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) can generate code from natural language, but the extent to which they capture intended program behavior remains unclear. Executable behavioral specifications, defined via preconditions and postconditions, provide a concrete means to assess such understanding. However, existing work on specification generation is constrained in evaluation methodology, task settings, and specification expressiveness. We introduce CodeSpecBench, a benchmark for executable behavioral specification generation under an execution-based evaluation protocol. CodeSpecBench supports both function-level and repository-level tasks and encodes specifications as executable Python functions. Constructed from diverse real-world codebases, it enables a realistic assessment of both correctness (accepting valid behaviors) and completeness (rejecting invalid behaviors). Evaluating 15 state-of-the-art LLMs on CodeSpecBench, we observe a sharp performance degradation on repository-level tasks, where the best model attains only a 20.2% pass rate. We further find that specification generation is substantially more challenging than code generation, indicating that strong coding performance does not necessarily reflect deep understanding of intended program semantics. Our data and code are available at https://github.com/SparksofAGI/CodeSpecBench.

Related papers

• SpecCoder: Specification-Aware Code Generation with Curriculum Dual-Task Reinforcement Learning [30.588079618949447]
  SpecCoder is a specification-aware two-stage training framework for code generation.<n>It first employs specification-guided SFT to train GRPs to derive structured specification analyses and generate code conditioned on them.<n>It then introduces curriculum dual-taskO, which jointly optimize specification-guided generation and discrimination to encourage stronger correspondence between specifications and code behavior.
  arXiv  Detail & Related papers  (2026-09-05T11:52:54Z)
• BeSpec: Behavior-Level Specification Alignment for Code Generation [2.9611475619490633]
  This paper presents BeSpec, a behavioral model-based approach to specification alignment.<n>BeSpec treats the task description as partial evidence about the intended behavior of the correct program.<n>It generates candidate programs, executes them on probe inputs, and compares their observed behavior with the predicted behaviors.
  arXiv  Detail & Related papers  (2026-07-03T04:48:45Z)
• CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustments [1.3999481573773072]
  We introduce a multi-language benchmark that evaluates instruction-following capabilities and is to operate on any set of standalone coding problems.<n>Our benchmark evaluates instruction following in two key settings: adherence to pre-defined constraints specified with the initial problem, and the ability to perform refinements based on follow-up instructions.
  arXiv  Detail & Related papers  (2025-10-31T15:47:07Z)
• Gistify! Codebase-Level Understanding via Runtime Execution [97.63528391679807]
  We propose Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality.<n>We show that current state-of-the-art models struggle to reliably solve Gistify tasks, especially ones with long executions.
  arXiv  Detail & Related papers  (2025-10-30T17:58:26Z)
• IFEvalCode: Controlled Code Generation [69.28317223249358]
  The paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs.<n>The authors present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages.
  arXiv  Detail & Related papers  (2025-07-30T08:08:48Z)
• VERINA: Benchmarking Verifiable Code Generation [46.582574591358735]
  Large language models (LLMs) are increasingly integrated in software development.<n> Ensuring correctness in LLM-generated code remains challenging.<n>Verifiable code generation offers a promising path to address this limitation.
  arXiv  Detail & Related papers  (2025-05-29T06:12:52Z)
• ReCode: Robustness Evaluation of Code Generation Models [90.10436771217243]
  We propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. With human annotators, we verified that over 90% of the perturbed prompts do not alter the semantic meaning of the original prompt.
  arXiv  Detail & Related papers  (2022-12-20T14:11:31Z)
• Python Code Generation by Asking Clarification Questions [57.63906360576212]
  In this work, we introduce a novel and more realistic setup for this task. We hypothesize that the under-specification of a natural language description can be resolved by asking clarification questions. We collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers.
  arXiv  Detail & Related papers  (2022-12-19T22:08:36Z)
• ReACC: A Retrieval-Augmented Code Completion Framework [53.49707123661763]
  We propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.
  arXiv  Detail & Related papers  (2022-03-15T08:25:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.