Continuous Speculative Decoding for Autoregressive Image Generation

• URL: http://arxiv.org/abs/2411.11925v2
• Date: Sun, 28 Sep 2025 08:54:03 GMT
• Title: Continuous Speculative Decoding for Autoregressive Image Generation
• Authors: Zili Wang, Robert Zhang, Kun Ding, Qi Yang, Fei Li, Shiming Xiang,
• Abstract summary: Continuous visual autoregressive (AR) models have demonstrated promising performance in image generation.<n> speculative decoding has effectively accelerated discrete autoregressive inference.<n>This work addresses challenges from low acceptance rate, inconsistent output distribution, and modified distribution without analytic expression.
• Score: 27.308442169466975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous visual autoregressive (AR) models have demonstrated promising performance in image generation. However, the heavy autoregressive inference burden imposes significant overhead. In Large Language Models (LLMs), speculative decoding has effectively accelerated discrete autoregressive inference. However, the absence of an analogous theory for continuous distributions precludes its use in accelerating continuous AR models. To fill this gap, this work presents continuous speculative decoding, and addresses challenges from: 1) low acceptance rate, caused by inconsistent output distribution between target and draft models, and 2) modified distribution without analytic expression, caused by complex integral. To address challenge 1), we propose denoising trajectory alignment and token pre-filling strategies. To address challenge 2), we introduce acceptance-rejection sampling algorithm with an appropriate upper bound, thereby avoiding explicitly calculating the integral. Furthermore, our denoising trajectory alignment is also reused in acceptance-rejection sampling, effectively avoiding repetitive diffusion model inference. Extensive experiments demonstrate that our proposed continuous speculative decoding achieves over $2\times$ speedup on off-the-shelf models, while maintaining the original generation quality. Codes is available at: https://github.com/MarkXCloud/CSpD

Related papers

• $\f{D^3}$QE: Learning Discrete Distribution Discrepancy-aware Quantization Error for Autoregressive-Generated Image Detection [85.9202830503973]
  Visual autoregressive (AR) models generate images through discrete token prediction.<n>We propose to leverage Discrete Distribution Discrepancy-aware Quantization Error (D$3$QE) for autoregressive-generated image detection.
  arXiv  Detail & Related papers  (2025-10-07T13:02:27Z)
• Test-Time Anchoring for Discrete Diffusion Posterior Sampling [38.507644561076894]
  Posterior sampling is a challenging problem for pretrained discrete diffusion foundation models.<n>We introduce Anchored Posterior Sampling (APS) for masked diffusion foundation models.<n>Our approach achieves state-of-the-art performance among discrete diffusion samplers across linear and nonlinear inverse problems.
  arXiv  Detail & Related papers  (2025-10-02T17:58:37Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Rethinking Discrete Tokens: Treating Them as Conditions for Continuous Autoregressive Image Synthesis [79.98107530577576]
  DisCon is a novel framework that reinterprets discrete tokens as conditional signals rather than generation targets.<n>DisCon achieves a gFID score of 1.38 on ImageNet 256$times $256 generation, outperforming state-of-the-art autoregressive approaches by a clear margin.
  arXiv  Detail & Related papers  (2025-07-02T14:33:52Z)
• Fast Autoregressive Models for Continuous Latent Generation [49.079819389916764]
  Autoregressive models have demonstrated remarkable success in sequential data generation, particularly in NLP. Recent work, the masked autoregressive model (MAR) bypasses quantization by modeling per-token distributions in continuous spaces using a diffusion head. We propose Fast AutoRegressive model (FAR), a novel framework that replaces MAR's diffusion head with a lightweight shortcut head.
  arXiv  Detail & Related papers  (2025-04-24T13:57:08Z)
• Frequency Autoregressive Image Generation with Continuous Tokens [31.833852108014312]
  We introduce the frequency progressive autoregressive (textbfFAR) paradigm and instantiate FAR with the continuous tokenizer.<n>We demonstrate the efficacy of FAR through comprehensive experiments on the ImageNet dataset.
  arXiv  Detail & Related papers  (2025-03-07T10:34:04Z)
• Distributional Diffusion Models with Scoring Rules [83.38210785728994]
  Diffusion models generate high-quality synthetic data. generating high-quality outputs requires many discretization steps. We propose to accomplish sample generation by learning the posterior em distribution of clean data samples.
  arXiv  Detail & Related papers  (2025-02-04T16:59:03Z)
• RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction [17.005198258689035]
  Diffusion Probabilistic Models (DPMs) have emerged as the de facto approach for high-fidelity image synthesis. We introduce a novel generative framework, the Recurrent Diffusion Probabilistic Model (RDPM), which enhances the diffusion process through a recurrent token prediction mechanism.
  arXiv  Detail & Related papers  (2024-12-24T12:28:19Z)
• Fast constrained sampling in pre-trained diffusion models [80.99262780028015]
  We propose an algorithm that enables fast, high-quality generation under arbitrary constraints.<n>Our approach produces results that rival or surpass the state-of-the-art training-free inference methods.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement [80.18490952057125]
  Iterative refinement has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) on complex tasks. We propose Context-Wise Order-Agnostic Language Modeling (COrAL) to overcome these challenges. Our approach models multiple token dependencies within manageable context windows, enabling the model to perform iterative refinement internally.
  arXiv  Detail & Related papers  (2024-10-12T23:56:19Z)
• LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding [30.630803933771865]
  Experimental results demonstrate the efficacy of our method in providing a substantial speed-up over speculative decoding. LANTERN increases speed-ups by $mathbf1.75times$ and $mathbf1.76times$, as compared to greedy decoding and random sampling.
  arXiv  Detail & Related papers  (2024-10-04T12:21:03Z)
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion [61.03681839276652]
  Diffusion Forcing is a new training paradigm where a diffusion model is trained to denoise a set of tokens with independent per-token noise levels. We apply Diffusion Forcing to sequence generative modeling by training a causal next-token prediction model to generate one or several future tokens.
  arXiv  Detail & Related papers  (2024-07-01T15:43:25Z)
• Autoregressive Image Generation without Vector Quantization [31.798754606008067]
  Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens. We propose to model the per-token probability distribution using a diffusion procedure, which allows us to apply autoregressive models in a continuous-valued space.
  arXiv  Detail & Related papers  (2024-06-17T17:59:58Z)
• Iterative Token Evaluation and Refinement for Real-World Super-Resolution [77.74289677520508]
  Real-world image super-resolution (RWSR) is a long-standing problem as low-quality (LQ) images often have complex and unidentified degradations. We propose an Iterative Token Evaluation and Refinement framework for RWSR. We show that ITER is easier to train than Generative Adversarial Networks (GANs) and more efficient than continuous diffusion models.
  arXiv  Detail & Related papers  (2023-12-09T17:07:32Z)
• Complexity Matters: Rethinking the Latent Space for Generative Modeling [65.64763873078114]
  In generative modeling, numerous successful approaches leverage a low-dimensional latent space, e.g., Stable Diffusion. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity.
  arXiv  Detail & Related papers  (2023-07-17T07:12:29Z)
• Variational Diffusion Auto-encoder: Latent Space Extraction from Pre-trained Diffusion Models [0.0]
  Variational Auto-Encoders (VAEs) face challenges with the quality of generated images, often presenting noticeable blurriness. This issue stems from the unrealistic assumption that approximates the conditional data distribution, $p(textbfx | textbfz)$, as an isotropic Gaussian. We illustrate how one can extract a latent space from a pre-existing diffusion model by optimizing an encoder to maximize the marginal data log-likelihood.
  arXiv  Detail & Related papers  (2023-04-24T14:44:47Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)
• ChiroDiff: Modelling chirographic data with Diffusion Models [132.5223191478268]
  We introduce a powerful model-class namely "Denoising Diffusion Probabilistic Models" or DDPMs for chirographic data. Our model named "ChiroDiff", being non-autoregressive, learns to capture holistic concepts and therefore remains resilient to higher temporal sampling rate.
  arXiv  Detail & Related papers  (2023-04-07T15:17:48Z)
• Generation of data on discontinuous manifolds via continuous stochastic non-invertible networks [6.201770337181472]
  We show how to generate discontinuous distributions using continuous networks. We derive a link between the cost functions and the information-theoretic formulation. We apply our approach to synthetic 2D distributions to demonstrate both reconstruction and generation of discontinuous distributions.
  arXiv  Detail & Related papers  (2021-12-17T17:39:59Z)
• Symbolic Music Generation with Diffusion Models [4.817429789586127]
  We present a technique for training diffusion models on sequential data by parameterizing the discrete domain in the continuous latent space of a pre-trained variational autoencoder. We show strong unconditional generation and post-hoc conditional infilling results compared to autoregressive language models operating over the same continuous embeddings.
  arXiv  Detail & Related papers  (2021-03-30T05:48:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.