AdapTrack: Constrained Decoding without Distorting LLM's Output Intent

• URL: http://arxiv.org/abs/2510.17376v1
• Date: Mon, 20 Oct 2025 10:11:34 GMT
• Title: AdapTrack: Constrained Decoding without Distorting LLM's Output Intent
• Abstract summary: Language model-based code generation and completion tools can sometimes produce code that does not meet necessary constraints.<n> Constrained decoding techniques are developed to help the model generate code adhering to the constraints.<n>AdapTrack avoids distorting the output intent of the model, thereby producing results that are not only constraint-compliant but also semantically aligned with model's output intent.
• Score: 53.75852524070165
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language model-based code generation and completion tools have been widely adopted, but they may sometimes produce code that does not meet necessary constraints, such as syntactic correctness or API existence. Constrained decoding techniques are developed to help the model generate code adhering to the constraints by greedily eliminating generation options that violate constraints at each step of the generation process. However, there is a severe limitation of constrained decoding, that it distorts the model's output intent, forcing it to produce code that may satisfy the constraint but does not match the development intent and is therefore incorrect. In response to this challenge, we propose AdapTrack. By incorporating backtracking into the generation process, AdapTrack avoids distorting the output intent of the model, thereby producing results that are not only constraint-compliant but also more semantically aligned with model's output intent. On our synthetic API completion dataset, AdapTrack can achieve up to 360.87% improvement compared to constrained decoding; on the real-world API completion dataset we collect that exhibits similar issues, AdapTrack can achieve up to 38.93% improvement over constrained decoding; in general code genration benchmarks, compared to constrained decoding, AdapTrack can achieve up to 7.84% improvement on HumanEval, and up to 6.42% improvement on MBPP. This indicates that, simply by better adhering to the model's output intent, AdapTrack can achieve significant improvements. We provide a theoretical proof that the distribution produced by AdapTrack aligns with the model's distribution given the generated tokens, thereby ensuring that the model's output intent is not distorted. Experiments on DSL problems show that, compared to existing methods, our approach can provide generation results that are more consistent with the language model's distribution.

Related papers

• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• Encoder-Decoder Manifold Alignment for Idempotent Generation [51.959208893243776]
  We argue that a key reason for this failure is a geometric mismatch between the manifold learned by the encoder and decoder.<n>We propose a new training framework that explicitly closes this gap by forcing the encoder and decoder to learn consistent representations of the same underlying data manifold.
  arXiv  Detail & Related papers  (2026-06-21T02:14:37Z)
• EPIC: Efficient and Parallel Inference under CFG Constraints for Diffusion Language Models [9.328325284438595]
  Current methods can be up to four times slower than unconstrained decoding.<n>We propose an efficient CFG-constrained decoding framework, EPIC, that addresses this limitation.<n>Our method improves decoding efficiency by combining lexing memoization, validation using Earley-style parsing instead of deterministic automata, and relaxed compatible subset selection for parallel commit.
  arXiv  Detail & Related papers  (2026-05-30T13:20:03Z)
• DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification [29.426184837710952]
  Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel.<n>We propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality.<n>We demonstrate that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods.
  arXiv  Detail & Related papers  (2026-04-08T21:52:32Z)
• Continuous Diffusion Models Can Obey Formal Syntax [37.090659119764176]
  Diffusion language models offer a promising alternative to autoregressive models due to their global, non-causal process.<n>We introduce a training-free method for steering continuous language models to satisfy formal syntactic constraints expressed using regular expressions.<n>Our approach constructs an analytic score estimating the probability that a latent state decodes to a valid string accepted by a given regular expression.
  arXiv  Detail & Related papers  (2026-02-12T22:55:05Z)
• Draft-Conditioned Constrained Decoding for Structured Generation in LLMs [11.309525632171217]
  Constrained decoding can distort generation when the model assigns low probability mass to valid continuations.<n>We propose emphDraft-Conditioned Constrained Decoding (DCCD), a training-free inference procedure that decouples semantic planning from structural enforcement.<n>We analyze DCCD through a KL-projection view, showing that draft conditioning increases feasible mass and reduces the cumulative "projection tax" induced by hard constraints.
  arXiv  Detail & Related papers  (2026-02-08T03:52:24Z)
• IntentCoding: Amplifying User Intent in Code Generation [66.86062682889101]
  Large Language Models (LLMs) have shown strong capabilities in code generation, but their adherence to fine-grained user intent with multiple constraints remains a significant challenge.<n>We propose Intent-Amplified Code Generation (IntentCoding), a novel decoding strategy that enhances an LLM's ability to follow user intent.<n>IntentCoding captures the influence of user intent by masking out the intent, and applies a multi-strength ensemble mechanism to amplify the effect of user intent during generation.
  arXiv  Detail & Related papers  (2026-01-20T13:34:16Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation [69.62857948698436]
  Recent advances in large language models (LLMs) have improved their performance on coding benchmarks.<n>However, improvement is plateauing due to the exhaustion of readily available high-quality data.<n>We propose Sol-Ver, a self-play solver-verifier framework that jointly improves a single model's code and test generation capacity.
  arXiv  Detail & Related papers  (2025-02-20T18:32:19Z)
• Continuous Speculative Decoding for Autoregressive Image Generation [27.308442169466975]
  Continuous visual autoregressive (AR) models have demonstrated promising performance in image generation.<n> speculative decoding has effectively accelerated discrete autoregressive inference.<n>This work addresses challenges from low acceptance rate, inconsistent output distribution, and modified distribution without analytic expression.
  arXiv  Detail & Related papers  (2024-11-18T09:19:15Z)
• COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement [80.18490952057125]
  Iterative refinement has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) on complex tasks. We propose Context-Wise Order-Agnostic Language Modeling (COrAL) to overcome these challenges. Our approach models multiple token dependencies within manageable context windows, enabling the model to perform iterative refinement internally.
  arXiv  Detail & Related papers  (2024-10-12T23:56:19Z)
• Truth or Deceit? A Bayesian Decoding Game Enhances Consistency and Reliability [10.381674636820025]
  Large Language Models (LLMs) often produce outputs that lack consistency and reliability. We propose a novel game-theoretic approach to enhance consistency and reliability during the decoding stage of LLM output generation.
  arXiv  Detail & Related papers  (2024-10-01T20:46:10Z)
• FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking [57.53742155914176]
  API call generation is the cornerstone of large language models' tool-using ability. Existing supervised and in-context learning approaches suffer from high training costs, poor data efficiency, and generated API calls that can be unfaithful to the API documentation and the user's request. We propose an output-side optimization approach called FANTASE to address these limitations.
  arXiv  Detail & Related papers  (2024-07-18T23:44:02Z)
• Online Speculative Decoding [34.987825705622555]
  We introduce online speculative decoding to accelerate the inference of large language models. The main idea is to continuously update the (multiple) draft model(s) on observed user query data. We develop a prototype of online speculative decoding based on knowledge distillation and evaluate it using both synthetic and real query data.
  arXiv  Detail & Related papers  (2023-10-11T04:03:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.