CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing

• URL: http://arxiv.org/abs/2608.13925v1
• Date: Fri, 14 Aug 2026 03:52:33 GMT
• Title: CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing
• Abstract summary: Diffusion large language models (dLLMs) accelerate language generation by predicting multiple masks in a single forward pass.<n>Existing dLLMs can suffer from unreliable predictions in early denoising stages under aggressive parallelism strategies.<n>We present Consistency Forcing (CForce) for dLLMs, a distillation method to force the mask predictions of early stages to align with those of later stages.
• Score: 30.73559220728677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion large language models (dLLMs) accelerate language generation by predicting multiple masks in a single forward pass. However, existing dLLMs can suffer from unreliable predictions in early denoising stages under aggressive parallelism strategies, leading to errors that can propagate to later stages. To tackle this issue, we present Consistency Forcing (CForce) for dLLMs, a distillation method to force the mask predictions of early stages to align with those of later stages. CForce trains the model on pre-collected self-rollout trajectories, thereby improving training-inference alignment. We introduce Confidence Adaptive KL Divergence as a distillation objective to conjoin the merits of forward and reverse KL. We further provide a theoretical analysis for the consistency objective to explain why CForce can approximately minimize the prediction error of early stages. Critically, the same formulation applies to both mask-to-token decoding and edit-capable decoding; in the edit-capable case, later token-to-token refinements provide additional supervision for earlier masked-state predictions. Experiments on non-edit and edit-capable LLaDA models show improved speed-quality trade-offs, especially under high-parallelism decoding budgets. Code is available at: https://github.com/inclusionAI/dFactory.

Related papers

• Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding [92.1521161575198]
  We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer.<n> Experiments across dense and Mixture-of-Experts LLMs demonstrate consistent gains on challenging reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-06-20T07:03:26Z)
• SimSD: Simple Speculative Decoding in Diffusion Language Models [61.33773959352141]
  Diffusion large language models (dLLMs) offer faster inference through parallel or blockwise decoding.<n>Their masked language modeling formulation remains incompatible with standard token-level speculative decoding.<n>We propose a speculative decoding algorithm for dLLMs, named SimSD, that equips dLLMs with temporally valid token-level contexts.<n>Our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
  arXiv  Detail & Related papers  (2026-06-01T17:46:46Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection [35.2079721755684]
  We introduce LEAP (Lookahead Early-Convergence Token Detection for Accelerated Parallel Decoding)<n>LEAP is a training-free, plug-and-play method that leverages future context filtering and multi-sequence superposition to detect early-converging tokens.<n>On the GSM8K dataset, combining LEAP with dParallel accelerates decoding to 7.2 tokens per step while preserving model precision.
  arXiv  Detail & Related papers  (2026-05-09T03:26:23Z)
• Teach Diffusion Language Models to Learn from Their Own Mistakes [45.68746718883178]
  Masked Diffusion Language Models (DLMs) achieve significant speed by generating multiple tokens in parallel.<n> parallel sampling approach will introduce strong dependency errors and cause quality to deteriorate rapidly as the generation step size grows.<n>We propose Decoupled Self-Correction to maintain high-quality multi-token generation.
  arXiv  Detail & Related papers  (2026-01-10T05:04:33Z)
• Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models [0.0]
  Large Language Models (DLLMs) enable fully parallel token decoding but often remain impractical at inference time.<n>Most existing acceleration methods focus on traversing this generative trajectory more efficiently via improved solvers or sampling strategies.<n>We propose a training-free interface that injects prompt-conditioned priors from a lightweight auxiliary model into the diffusion initialization.<n>Because injected priors can be imperfect and unmask-only decoding can over-commit early, we also introduce a simple confidence-based remasking mechanism as a form of prior skepticism.
  arXiv  Detail & Related papers  (2025-12-22T03:45:04Z)
• Continuous Autoregressive Language Models [56.49239051750678]
  We introduce Continuous Autoregressive Language Models (CALM)<n>CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector.<n>We develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling.
  arXiv  Detail & Related papers  (2025-10-31T17:58:11Z)
• Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States [28.663951525871756]
  We introduce Latent Refinement Decoding (LRD), a two-stage framework with Latent Refinement and a Predictive Feedback Loop.<n>LRD improves accuracy while delivering speedups of up to 10.6x, making it a strong and versatile alternative for parallel sequence generation.
  arXiv  Detail & Related papers  (2025-10-13T06:38:13Z)
• Diffusion Language Models Know the Answer Before Decoding [56.96815863705218]
  Diffusion language models (DLMs) have emerged as an alternative to autoregressive approaches.<n>Our work highlights and leverage an overlooked property of DLMs early answer convergence.<n>We introduce Prophet, a training-free fast decoding paradigm that enables early commit decoding.
  arXiv  Detail & Related papers  (2025-08-27T15:40:25Z)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding [60.407727995313074]
  We introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel.<n>APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-31T06:10:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.