The Alignment Problem in Constrained Code Generation

• URL: http://arxiv.org/abs/2606.21619v1
• Date: Fri, 19 Jun 2026 17:22:27 GMT
• Title: The Alignment Problem in Constrained Code Generation
• Abstract summary: We study how language models react to varying levels of incompleteness.<n>Unconstrained decoding significantly outperforms constrained decoding in terms of functional correctness.<n>These contributions make the community aware of the negative effects of misalignment in constrained decoding.
• Score: 5.219568203653523
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in code generation, but their outputs frequently contain syntax or type errors that result in compilation failures. Constrained decoding has been proposed as a solution to mitigate compilation errors by construction, improving functional correctness as a byproduct. However, previous works overlook a critical aspect of constrained decoding: the alignment between constrainer (e.g., types), language model and the target specification language (e.g., TypeScript). Misalignment is caused by the constrainer being incomplete--rejecting programs that belong to the target--or unsound--allowing programs that are not part of the target. The bias created by incompleteness distorts the language model distribution, and can be detrimental for code generation. We evaluate this hypothesis using seven language models, two target languages, two constrainers, enforcing types and syntax during decoding, and we study how language models react to varying levels of incompleteness. On three benchmarks, when the constrainer is incomplete, unconstrained decoding significantly outperforms constrained decoding in terms of functional correctness. Incompleteness pushes the model into low-probability regions of the program space, causing the generation to frequently time out, and reducing functional correctness by up to 97%. These contributions make the community aware of the negative effects of misalignment in constrained decoding, and provide quantitative insights on how to design constrainers that are beneficial for code generation systems with formal guarantees.

Related papers

• Readability-Robust Code Summarization via Meta Curriculum Learning [53.44612630063336]
  In the real world, code is often poorly structured or obfuscated, significantly degrading model performance.<n>We propose RoFTCodeSum, a novel fine-tuning method that enhances the robustness of code summarization against poorly readable code.
  arXiv  Detail & Related papers  (2026-01-09T02:38:24Z)
• Type-Constrained Code Generation with Language Models [51.03439021895432]
  We introduce a type-constrained decoding approach that leverages type systems to guide code generation.<n>For this purpose, we develop novel prefix automata and a search over inhabitable types, forming a sound approach to enforce well-typedness on LLM-generated code.<n>Our approach reduces compilation errors by more than half and significantly increases functional correctness in code synthesis, translation, and repair tasks.
  arXiv  Detail & Related papers  (2025-04-12T15:03:00Z)
• Galapagos: Automated N-Version Programming with LLMs [6.745049148887976]
  We propose the automated generation of program variants using large language models.<n>We design, develop and evaluate Gal'apagos: a tool for generating program variants.<n>We evaluate Gal'apagos by creating N-Version components of real-world C code.
  arXiv  Detail & Related papers  (2024-08-18T16:44:01Z)
• Supporting Cross-language Cross-project Bug Localization Using Pre-trained Language Models [2.5121668584771837]
  Existing techniques often struggle with generalizability and deployment due to their reliance on application-specific data. This paper proposes a novel pre-trained language model (PLM) based technique for bug localization that transcends project and language boundaries.
  arXiv  Detail & Related papers  (2024-07-03T01:09:36Z)
• Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs [57.27982780697922]
  Large language models have demonstrated exceptional capability in natural language understanding and generation. However, their generation speed is limited by the inherently sequential nature of their decoding process. This paper introduces Lexical Unit Decoding, a novel decoding methodology implemented in a data-driven manner.
  arXiv  Detail & Related papers  (2024-05-24T04:35:13Z)
• Constrained Decoding for Fill-in-the-Middle Code Language Models via Efficient Left and Right Quotienting of Context-Sensitive Grammars [11.279507894576213]
  This paper contributes an incremental synthesis that allows early rejection of syntactically incorrect code. We extend the Earley parsing algorithm to allow for left and right quotients of context-free grammars.
  arXiv  Detail & Related papers  (2024-02-28T02:12:47Z)
• Code Representation Learning At Scale [75.04686476303436]
  We fuel code representation learning with a vast amount of code data via a two-stage pretraining scheme. We first train the encoders via a mix that leverages both randomness in masking language modeling and the structure aspect of programming language. We then enhance the representations via contrastive learning with hard negative and hard positive constructed in an unsupervised manner.
  arXiv  Detail & Related papers  (2024-02-02T22:19:15Z)
• Teaching Large Language Models to Self-Debug [62.424077000154945]
  Large language models (LLMs) have achieved impressive performance on code generation. We propose Self- Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.
  arXiv  Detail & Related papers  (2023-04-11T10:43:43Z)
• Generating Sequences by Learning to Self-Correct [64.0249217590888]
  Self-Correction decouples an imperfect base generator from a separate corrector that learns to iteratively correct imperfect generations. We show that Self-Correction improves upon the base generator in three diverse generation tasks.
  arXiv  Detail & Related papers  (2022-10-31T18:09:51Z)
• FixEval: Execution-based Evaluation of Program Fixes for Programming Problems [23.987104440395576]
  We introduce FixEval, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixEval offers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately.
  arXiv  Detail & Related papers  (2022-06-15T20:18:43Z)
• Compilable Neural Code Generation with Compiler Feedback [43.97362484564799]
  This paper proposes a three-stage pipeline for compilable code generation, including language model fine-tuning, compilability reinforcement, and compilability discrimination. Experiments on two code generation tasks demonstrate the effectiveness of our proposed approach, improving the success rate of compilation from 44.18 to 89.18 on average and from 70.3 to 96.2 in text-to-code generation, respectively.
  arXiv  Detail & Related papers  (2022-03-10T03:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.