Large Language Continuous Diffusion Models
Abstract Overview
The paper introduces Sigma, a family of 3B and 8B continuous diffusion language models that operate over steerable 16-dimensional latent trajectories rather than discrete token states. Sigma is trained blockwise with a likelihood-based objective, jointly learning token embedding geometry while denoising Gaussian-corrupted embeddings, and it is warm-started from autoregressive checkpoints with AR regularization to improve optimization stability. On the inference side, the study analyzes blockwise DDPM and DDIM sampling combined with embedding-space classifier-free guidance, score temperature, and self-conditioning. The model is evaluated on mathematical reasoning and coding benchmarks in both base and supervised fine-tuned settings, alongside investigations into few-step generation, spherical embedding regularization, and parallel decoding distillation.
Novelty
The main novelty is scaling continuous diffusion language modeling to 3B and 8B parameters for conditional reasoning and coding using low-dimensional learned continuous trajectories. The work also introduces an inference steering framework utilizing embedding-space classifier-free guidance and score temperature, complemented by spherical embedding regularization and parallel decoding distillation tailored for continuous dLMs.
Results
On base-model benchmarks, Sigma-8B with DDPM achieves 89.07% on GSM8K, 67.47% on Minerva Math, and a 69.67% average across six tasks, matching state-of-the-art discrete diffusion baselines like Nemotron-Labs-Diffusion-8B (70.22%). In pass@k evaluations on MBPP, Sigma-3B surpasses Nemotron-Labs-Diffusion after k=32, reaching 91.8% at k=256 compared to 87.6%. Additionally, combining spherical embedding regularization with parallel decoding distillation significantly improves low-NFE generation at 4 to 8 steps per block.
Key Points
- Sigma is introduced as the first large-scale (3B/8B) continuous diffusion language model family, operating over learned 16D continuous latent trajectories and trained blockwise via likelihood optimization.
- Embedding-space classifier-free guidance and score temperature provide effective inference steering to control the quality-diversity trade-off with minimal computational overhead.
- Empirical evaluations show competitive reasoning and coding performance relative to discrete diffusion baselines, with enhanced few-step robustness via spherical embedding regularization and parallel decoding distillation.