Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion

• URL: http://arxiv.org/abs/2601.13599v2
• Date: Wed, 21 Jan 2026 18:21:39 GMT
• Title: Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion
• Abstract summary: We propose a 'draft-then-refine' framework to overcome the irreversibility and myopia problems inherent in block diffusion models.<n>Our approach sets a new benchmark for discrete diffusion models on the OpenWebText dataset.
• Score: 26.45111031153368
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: One of the most compelling features of global discrete diffusion language models is their global bidirectional contextual capability. However, existing block-based diffusion studies tend to introduce autoregressive priors, which, while offering benefits, can cause models to lose this global coherence at the macro level. To regain global contextual understanding while preserving the advantages of the semi-autoregressive paradigm, we propose Diffusion in Diffusion, a 'draft-then-refine' framework designed to overcome the irreversibility and myopia problems inherent in block diffusion models. Our approach first employs block diffusion to generate rapid drafts using small blocks, then refines these drafts through global bidirectional diffusion with a larger bidirectional receptive field. We utilize snapshot confidence remasking to identify the most critical tokens that require modification, and apply mix-scale training to expand the block diffusion model's global capabilities. Empirical results demonstrate that our approach sets a new benchmark for discrete diffusion models on the OpenWebText dataset. Using only 26% of the fine-tuning budget of baseline models, we reduce generative perplexity from 25.7 to 21.9, significantly narrowing the performance gap with autoregressive models.

Related papers

• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention [58.484607892210015]
  Diffusion Language Models (DLMs) enable globally coherent, bidirectional, and controllable text generation.<n>Many existing block-sparse attention methods select blocks by fixed sampling patterns over the high-resolution attention space.<n>We propose the Block Approximate Sparse Attention framework (BA-Att) with block-wise pre-downsampled operation.
  arXiv  Detail & Related papers  (2026-05-19T12:01:13Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation [32.945437908689286]
  We introduce MADFormer, a Mixed Autoregressive Diffusion and Transformer that serves as a testbed for analyzing AR-diffusion trade-offs.<n>We identify two key insights: (1) block-wise partitioning significantly improves performance on high-resolution images, and (2) vertically mixing AR and diffusion layers yields better quality-efficiency balances--improving FID by up to 75% under constrained inference compute.
  arXiv  Detail & Related papers  (2025-06-09T17:59:01Z)
• Consistent World Models via Foresight Diffusion [56.45012929930605]
  We argue that a key bottleneck in learning consistent diffusion-based world models lies in the suboptimal predictive ability.<n>We propose Foresight Diffusion (ForeDiff), a diffusion-based world modeling framework that enhances consistency by decoupling condition understanding from target denoising.
  arXiv  Detail & Related papers  (2025-05-22T10:01:59Z)
• CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation [13.250999667915254]
  Diffusion-based language models have emerged as a compelling alternative due to their powerful parallel generation capabilities and inherent editability.<n>We propose CtrlDiff, a dynamic and controllable semi-autoregressive framework that adaptively determines the size of each generation block based on local semantics.
  arXiv  Detail & Related papers  (2025-05-20T14:52:41Z)
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models [15.853201399662344]
  Diffusion language models offer unique benefits over autoregressive models.<n>They lag in likelihood modeling and are limited to fixed-length generation.<n>We introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models.
  arXiv  Detail & Related papers  (2025-03-12T17:43:40Z)
• Generalized Interpolating Discrete Diffusion [65.74168524007484]
  Masked diffusion is a popular choice due to its simplicity and effectiveness.<n>We generalize a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes.<n>Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality.
  arXiv  Detail & Related papers  (2025-03-06T14:30:55Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models [100.53662473219806]
  Diffusion-of-Thought (DoT) is a novel approach that integrates diffusion models with Chain-of-Thought.<n>DoT allows reasoning steps to diffuse over time through a diffusion language model.<n>Our results demonstrate the effectiveness of DoT in multi-digit multiplication, logic, and grade school math problems.
  arXiv  Detail & Related papers  (2024-02-12T16:23:28Z)
• Guided Diffusion from Self-Supervised Diffusion Features [49.78673164423208]
  Guidance serves as a key concept in diffusion models, yet its effectiveness is often limited by the need for extra data annotation or pretraining. We propose a framework to extract guidance from, and specifically for, diffusion models.
  arXiv  Detail & Related papers  (2023-12-14T11:19:11Z)
• Lipschitz Singularities in Diffusion Models [64.28196620345808]
  Diffusion models often display the infinite Lipschitz property of the network with respect to time variable near the zero point.<n>We propose a novel approach, dubbed E-TSDM, which alleviates the Lipschitz singularities of the diffusion model near the zero point.<n>Our work may advance the understanding of the general diffusion process, and also provide insights for the design of diffusion models.
  arXiv  Detail & Related papers  (2023-06-20T03:05:28Z)
• David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs [49.822063966687175]
  Diffusion-based language models are emerging as a promising alternative to autoregressive LMs. We propose methods to scale a recently proposed diffusion model SSD-LM from 0.4B to 13B parameters. We show that SSD-2 facilitates novel ensembles with 100x smaller models that can be customized and deployed by individual users.
  arXiv  Detail & Related papers  (2023-05-24T06:22:14Z)
• DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery [20.787180028571694]
  DiffusionSeg is a synthesis-exploitation framework containing two-stage strategies. We synthesize abundant images, and propose a novel training-free AttentionCut to obtain masks in the first stage. In the second exploitation stage, to bridge the structural gap, we use the inversion technique, to map the given image back to diffusion features.
  arXiv  Detail & Related papers  (2023-03-17T07:47:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.