Guiding a Diffusion Transformer with the Internal Dynamics of Itself

• URL: http://arxiv.org/abs/2512.24176v1
• Date: Tue, 30 Dec 2025 12:16:46 GMT
• Title: Guiding a Diffusion Transformer with the Internal Dynamics of Itself
• Abstract summary: Internal Guidance (IG) yields significant improvements in both training efficiency and generation quality on various baselines.<n>On ImageNet 256x256, SiT-XL/2+IG achieves FID=5.31 and FID=1.75 at 80 and 800 epochs.<n>On LightningDiT-XL/1+IG, LightningDiT-XL/1+IG achieves FID=1.34 which achieves a large margin between all of these methods.
• Score: 29.825583753955485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The diffusion model presents a powerful ability to capture the entire (conditional) data distribution. However, due to the lack of sufficient training and data to learn to cover low-probability areas, the model will be penalized for failing to generate high-quality images corresponding to these areas. To achieve better generation quality, guidance strategies such as classifier free guidance (CFG) can guide the samples to the high-probability areas during the sampling stage. However, the standard CFG often leads to over-simplified or distorted samples. On the other hand, the alternative line of guiding diffusion model with its bad version is limited by carefully designed degradation strategies, extra training and additional sampling steps. In this paper, we proposed a simple yet effective strategy Internal Guidance (IG), which introduces an auxiliary supervision on the intermediate layer during training process and extrapolates the intermediate and deep layer's outputs to obtain generative results during sampling process. This simple strategy yields significant improvements in both training efficiency and generation quality on various baselines. On ImageNet 256x256, SiT-XL/2+IG achieves FID=5.31 and FID=1.75 at 80 and 800 epochs. More impressively, LightningDiT-XL/1+IG achieves FID=1.34 which achieves a large margin between all of these methods. Combined with CFG, LightningDiT-XL/1+IG achieves the current state-of-the-art FID of 1.19.

Related papers

• A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples [53.8898272536272]
  Pixel-space diffusion models aim to learn an end-to-end generator directly over raw pixels.<n>Recent work improves pixel diffusion through alternative prediction targets, training objectives, and architectures.<n>We show there is a cheaper, complementary strategy: textbfa frozen, pretrained pixel diffusion model can guide itself
  arXiv  Detail & Related papers  (2026-07-31T07:52:08Z)
• Enhanced Low-Density Region Exploration in Classifier-Guided Diffusion Models Through Modified Reverse Diffusion Sampling [5.191572579366909]
  We propose a sampling-time, density-aware extension of conditional diffusion model.<n>We modify the guided reverse dynamics by steering trajectories toward low-confidence regions.<n>We show that the proposed sampler consistently improves ADM model recall at 64x64 resolution while maintaining a comparable FID.
  arXiv  Detail & Related papers  (2026-06-11T13:35:53Z)
• Momentum Guidance: Plug-and-Play Guidance for Flow Models [11.47531308210865]
  We introduce Momentum Guidance (MG), a new dimension of guidance that leverages the ODE trajectory itself.<n>MG extrapolates the current velocity using an exponential moving average of past velocities and preserves the standard one-evaluation-per-step cost.<n>MG achieves average improvements in FID of 36.68% without CFG and 25.52% with CFG across various sampling settings, attaining an FID of 1.597 at 64 sampling steps.
  arXiv  Detail & Related papers  (2026-02-23T21:06:35Z)
• RS-Prune: Training-Free Data Pruning at High Ratios for Efficient Remote Sensing Diffusion Foundation Models [14.093802378976315]
  Diffusion-based remote sensing (RS) generative foundation models rely on large amounts of globally representative data.<n>We propose a training-free, two-stage data pruning approach that quickly select a high-quality subset under high pruning ratios.<n> Experiments show that, even after pruning 85% of the training data, our method significantly improves convergence and generation quality.
  arXiv  Detail & Related papers  (2025-12-29T06:44:06Z)
• Few-Step Diffusion via Score identity Distillation [67.07985339442703]
  Diffusion distillation has emerged as a promising strategy for accelerating text-to-image (T2I) diffusion models.<n>Existing methods rely on real or teacher-synthesized images to perform well when distilling high-resolution T2I diffusion models.<n>We propose two new guidance strategies: Zero-CFG, which disables CFG in the teacher and removes text conditioning in the fake score network, and Anti-CFG, which applies negative CFG in the fake score network.
  arXiv  Detail & Related papers  (2025-05-19T03:45:16Z)
• Flow-GRPO: Training Flow Matching Models via Online RL [80.62659379624867]
  We propose Flow-GRPO, the first method to integrate online policy reinforcement learning into flow matching models.<n>Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation into an equivalent Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps.
  arXiv  Detail & Related papers  (2025-05-08T17:58:45Z)
• Diffusion Models without Classifier-free Guidance [41.59396565229466]
  Model-guidance (MG) is a novel objective for training diffusion model addresses and removes commonly used guidance (CFG)<n>Our innovative approach transcends the standard modeling and incorporates the posterior probability of conditions.<n>Our method significantly accelerates the training process, doubles inference speed, and achieve exceptional quality that parallel surpass even concurrent diffusion models with CFG.
  arXiv  Detail & Related papers  (2025-02-17T18:59:50Z)
• Self-Guidance: Boosting Flow and Diffusion Generation on Their Own [35.56845917727121]
  Self-Guidance (SG) can significantly improve the quality of the generated image by suppressing the generation of low-quality samples.<n>SG relies on the sampling score function of the original diffusion or flow model at different noise levels.<n>We conduct extensive experiments on text-to-image and text-to-video generation with different architectures.
  arXiv  Detail & Related papers  (2024-12-08T06:32:27Z)
• Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think [72.48325960659822]
  One main bottleneck in training large-scale diffusion models for generation lies in effectively learning these representations.<n>We study this by introducing a straightforward regularization called REPresentation Alignment (REPA), which aligns the projections of noisy input hidden states in denoising networks with clean image representations obtained from external, pretrained visual encoders.<n>The results are striking: our simple strategy yields significant improvements in both training efficiency and generation quality when applied to popular diffusion and flow-based transformers, such as DiTs and SiTs.
  arXiv  Detail & Related papers  (2024-10-09T14:34:53Z)
• Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion [56.38386580040991]
  Consistency Trajectory Model (CTM) is a generalization of Consistency Models (CM) CTM enables the efficient combination of adversarial training and denoising score matching loss to enhance performance. Unlike CM, CTM's access to the score function can streamline the adoption of established controllable/conditional generation methods.
  arXiv  Detail & Related papers  (2023-10-01T05:07:17Z)
• LD-GAN: Low-Dimensional Generative Adversarial Network for Spectral Image Generation with Variance Regularization [72.4394510913927]
  Deep learning methods are state-of-the-art for spectral image (SI) computational tasks. GANs enable diverse augmentation by learning and sampling from the data distribution. GAN-based SI generation is challenging since the high-dimensionality nature of this kind of data hinders the convergence of the GAN training yielding to suboptimal generation. We propose a statistical regularization to control the low-dimensional representation variance for the autoencoder training and to achieve high diversity of samples generated with the GAN.
  arXiv  Detail & Related papers  (2023-04-29T00:25:02Z)
• Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data [125.7135706352493]
  Generative adversarial networks (GANs) typically require ample data for training in order to synthesize high-fidelity images. Recent studies have shown that training GANs with limited data remains formidable due to discriminator overfitting. This paper introduces a novel strategy called Adaptive Pseudo Augmentation (APA) to encourage healthy competition between the generator and the discriminator.
  arXiv  Detail & Related papers  (2021-11-12T18:13:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.