DiffusionGemma Technical Report

• URL: http://arxiv.org/abs/2608.00146v1
• Date: Fri, 31 Jul 2026 16:11:46 GMT
• Title: DiffusionGemma Technical Report
• Abstract summary: We introduce DiffusionGemma, a language model that uses discrete diffusion to generate text at exceptionally high speed.<n>Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel.<n>Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget.
• Score: 72.10570774996219
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

Related papers

• TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillation [105.14745366152725]
  TurboT2VA is a distillation and inference framework for joint text-to-video-audio generation.<n>On LTX-2, four-step distillation reduces generator latency from 50.52s to 2.51s at the standard evaluation resolution of 512$times$768.<n>The complete stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67$times$ generator-only speedup.
  arXiv  Detail & Related papers  (2026-08-25T15:09:02Z)
• ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models [22.63266813055847]
  We propose ReGen that jointly estimates multiple vector fields for both representations data within a single diffusion model.<n>We also present ReGenVoice, a latent diffusion model (LDM)-based text-to-speech model that achieves strong speech intelligibility.
  arXiv  Detail & Related papers  (2026-07-10T06:44:04Z)
• Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference [58.06027151683975]
  We present Seed Diffusion Preview, a large-scale language model based on discrete-state diffusion, offering remarkably fast inference speed.<n>Thanks to non-sequential, parallel generation, discrete diffusion models provide a notable speedup to mitigate the inherent latency of token-by-token decoding.
  arXiv  Detail & Related papers  (2025-08-04T08:43:01Z)
• Next Tokens Denoising for Speech Synthesis [51.320443764269726]
  Dragon-FM is a novel text-to-speech (TTS) design that unifies AR and flow-matching.<n>It processes 48 kHz audio tokens in chunks at a compact rate of 12.5 tokens per second.<n>Experiments on podcast datasets demonstrate its capability to efficiently generate high-quality zero-shot podcasts.
  arXiv  Detail & Related papers  (2025-07-30T15:03:36Z)
• FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion [22.207275433870937]
  Diffusion language models offer parallel token generation and inherent bidirectionality.<n>State-of-the-art diffusion models (e.g., Dream 7B, LLaDA 8B) suffer from slow inference.<n>We introduce Guided Diffusion, a training-free method that uses a lightweight pretrained autoregressive model to supervise token unmasking.
  arXiv  Detail & Related papers  (2025-05-27T17:39:39Z)
• ProReflow: Progressive Reflow with Decomposed Velocity [52.249464542399636]
  Flow matching aims to reflow the diffusion process of diffusion models into a straight line for a few-step and even one-step generation.<n>We introduce progressive reflow, which progressively reflows the diffusion models in local timesteps until the whole diffusion progresses.<n>We also introduce aligned v-prediction, which highlights the importance of direction matching in flow matching over magnitude matching.
  arXiv  Detail & Related papers  (2025-03-05T04:50:53Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [48.35054927704544]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• SparseDM: Toward Sparse Efficient Diffusion Models [20.783533300147866]
  We propose a method based on the improved Straight-Through Estimator to improve the deployment efficiency of diffusion models.<n> Experimental results on a Transformer and UNet-based diffusion models demonstrate that our method reduces MACs by 50% while maintaining FID.
  arXiv  Detail & Related papers  (2024-04-16T10:31:06Z)
• StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation [52.56469577812338]
  We introduce StreamDiffusion, a real-time diffusion pipeline for interactive image generation.<n>Existing diffusion models are adept at creating images from text or image prompts, yet they often fall short in real-time interaction.<n>We present a novel approach that transforms the original sequential denoising into the denoising process.
  arXiv  Detail & Related papers  (2023-12-19T18:18:33Z)
• Efficient Integrators for Diffusion Generative Models [22.01769257075573]
  Diffusion models suffer from slow sample generation at inference time. We propose two complementary frameworks for accelerating sample generation in pre-trained models. We present a hybrid method that leads to the best-reported performance for diffusion models in augmented spaces.
  arXiv  Detail & Related papers  (2023-10-11T21:04:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.