Looped Diffusion Transformer
Abstract Overview
This paper investigates scaling text-to-image diffusion models by repeatedly executing a shared group of Transformer blocks within each denoising step, thereby increasing computational depth without adding parameters. The authors observe that naive looping fails to reliably improve generation quality because of weak intermediate supervision and unmodulated attention updates that degrade spatial information. To solve these issues, they propose Looped Diffusion Transformer (Looped-DiT), which incorporates intermediate-loop deep supervision alongside self-modulating attention mechanisms. Comprehensive evaluations demonstrate improved parameter and inference efficiency, effective inference-time scaling, and latent visual reasoning behaviors such as progressive constraint satisfaction.
Novelty
The paper introduces Looped-DiT, presenting what it characterizes as the first systematic study of looped computation for text-to-image diffusion models. Its architectural novelty lies in combining intermediate-loop deep supervision with self-modulating attention (such as Exclusive Self Attention or Gated Attention) within shared looped Transformer blocks.
Results
Under matched-parameter and matched-compute settings, Looped-DiT consistently outperforms non-looped baselines, with a 260M Looped-DiT B/16 achieving an average score of 71.5 across six benchmarks to outperform models up to 6.5× larger while requiring 4.9× lower inference compute. Under fixed inference compute budgets, allocating computation to increasing loop depth yields greater performance improvements than allocating it to additional denoising steps. Furthermore, deep supervision enables flexible inference-time scaling and adaptive early-exit looping without retraining.
Key Points
- Naive looping within diffusion denoising steps suffers from diminishing or degrading returns due to weak intermediate supervision and loss of linearly decodable spatial position information across iterations.
- Looped-DiT stabilizes looped execution by decoding intermediate passes under flow-matching loss (Deep Supervision) and employing Self-Modulating Attention (such as XSA) to prevent excessive residual updates.
- The approach achieves higher parameter and compute efficiency than standard scaling or extra denoising steps, while exhibiting latent reasoning behaviors that resolve spatial constraints and correct errors across loops.