Generalized Discrete Diffusion with Self-Correction

• URL: http://arxiv.org/abs/2603.02230v1
• Date: Fri, 13 Feb 2026 04:12:35 GMT
• Title: Generalized Discrete Diffusion with Self-Correction
• Abstract summary: Self-correction is an effective technique for maintaining parallel sampling in discrete diffusion models with minimal performance degradation.<n>We propose a Self-Correcting Diffusion model to reformulate pretrained self-correction with explicit state transitions and learn directly in discrete time.<n>Our framework also simplifies the training noise schedule, eliminates a redundant remasking step, and relies exclusively on uniform transitions to learn self-correction.
• Score: 20.435906948993022
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-correction is an effective technique for maintaining parallel sampling in discrete diffusion models with minimal performance degradation. Prior work has explored self-correction at inference time or during post-training; however, such approaches often suffer from limited generalization and may impair reasoning performance. GIDD pioneers pretraining-based self-correction via a multi-step BERT-style uniform-absorbing objective. However, GIDD relies on a continuous interpolation-based pipeline with opaque interactions between uniform transitions and absorbing masks, which complicates hyperparameter tuning and hinders practical performance. In this work, we propose a Self-Correcting Discrete Diffusion (SCDD) model to reformulate pretrained self-correction with explicit state transitions and learn directly in discrete time. Our framework also simplifies the training noise schedule, eliminates a redundant remasking step, and relies exclusively on uniform transitions to learn self-correction. Experiments at the GPT-2 scale demonstrate that our method enables more efficient parallel decoding while preserving generation quality.

Related papers

• FlowBender: Feedback-Aware Training for Self-Correcting Conditional Flows [18.746098727212544]
  FlowBender is a new diffusion and flow model for 3D reconstruction.<n>We show that FlowBender consistently outperforms standard supervised alignment training.<n>We propose several variants of FlowBender for differentiable operators.
  arXiv  Detail & Related papers  (2026-06-18T15:56:07Z)
• Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction [7.637257579386074]
  Controllable generation with discrete diffusion models is often hindered by high computational overhead or the need for retraining.<n>We present underlinetextbfGradient-underlinetextbfInformed underlinetextbfCorrection (textbfGILC), a plug-and-play framework that efficiently estimates guidance signals.
  arXiv  Detail & Related papers  (2026-06-04T15:41:53Z)
• One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration [20.457786704615454]
  Fixed-Point Distillation (FPD) is an end-to-end framework that constructs local correction targets by partially corrupting the student's one-step draft and refining it with a single teacher step.<n>FPD achieves competitive visual fidelity and structural alignment within a single inference step, narrowing the gap to multi-step teachers.
  arXiv  Detail & Related papers  (2026-05-20T17:59:10Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training [54.8494905524997]
  Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes.<n>We propose SENTINEL, a verification mechanism for pipeline parallelism (PP) training without duplication.<n>Experiments demonstrate successful training of up to 4B- parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance.
  arXiv  Detail & Related papers  (2026-03-03T23:51:10Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• Authentic Discrete Diffusion Model [72.31371542619121]
  Authentic Discrete Diffusion (ADD) framework redefines prior pseudo-discrete approaches.<n>ADD reformulates the diffusion input by directly using float-encoded one-hot class data.<n> experiments demonstrate that ADD achieves superior performance on classification tasks compared to the baseline.
  arXiv  Detail & Related papers  (2025-10-01T15:51:10Z)
• ADT: Tuning Diffusion Models with Adversarial Supervision [16.974169058917443]
  Diffusion models have achieved outstanding image generation by reversing a forward noising process to approximate true data distributions.<n>We propose Adrial Diffusion Tuning (ADT) to stimulate the inference process during optimization and align the final outputs with training data.<n>ADT features a siamese-network discriminator with a fixed pre-trained backbone and lightweight trainable parameters.
  arXiv  Detail & Related papers  (2025-04-15T17:37:50Z)
• REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers [52.55041244336767]
  Traditional deep-learning wisdom dictates that end-to-end training is often preferable when possible.<n>For latent diffusion transformers, it is observed that end-to-end training both VAE and diffusion-model using standard diffusion-loss is ineffective.<n>We show that while diffusion loss is ineffective, end-to-end training can be unlocked through the representation-alignment (REPA) loss.
  arXiv  Detail & Related papers  (2025-04-14T17:59:53Z)
• Efficient Diffusion Training through Parallelization with Truncated Karhunen-Loève Expansion [5.770347328961063]
  Diffusion denoising models suffer from slow convergence during training.<n>We propose a novel forward-time process for training and sampling.<n>Our method significantly outperforms baseline diffusion models.
  arXiv  Detail & Related papers  (2025-03-22T05:34:02Z)
• Efficient Diffusion as Low Light Enhancer [63.789138528062225]
  Reflectance-Aware Trajectory Refinement (RATR) is a simple yet effective module to refine the teacher trajectory using the reflectance component of images. textbfReflectance-aware textbfDiffusion with textbfDistilled textbfTrajectory (textbfReDDiT) is an efficient and flexible distillation framework tailored for Low-Light Image Enhancement (LLIE)
  arXiv  Detail & Related papers  (2024-10-16T08:07:18Z)
• HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration [31.982294870690925]
  We develop a novel learning-based caching framework dubbed HarmoniCa.<n>It incorporates Step-Wise Denoising Training (SDT) to ensure the continuity of the denoising process.<n>Our framework achieves over $40%$ latency reduction (i.e., $2.07times$ theoretical speedup) and improved performance on PixArt-$alpha$.
  arXiv  Detail & Related papers  (2024-10-02T16:34:29Z)
• Dense Unsupervised Learning for Video Segmentation [49.46930315961636]
  We present a novel approach to unsupervised learning for video object segmentation (VOS) Unlike previous work, our formulation allows to learn dense feature representations directly in a fully convolutional regime. Our approach exceeds the segmentation accuracy of previous work despite using significantly less training data and compute power.
  arXiv  Detail & Related papers  (2021-11-11T15:15:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.