On the spectral properties of generative denoiser Jacobians
Abstract Overview
This paper studies the Jacobian of generative denoisers used in diffusion and flow-matching models, focusing on its spectrum as a way to characterize how denoisers differ beyond output-quality metrics. Across pre-trained ImageNet SiT models, the authors report that stronger generative performance is associated with larger principal Jacobian eigenvalues, especially in early and middle denoising steps and under classifier-free guidance. Motivated by this correlation, they introduce training regularizers based on input perturbations that either amplify responses along data-relevant directions or suppress responses along noisy directions without explicitly computing an eigendecomposition during training. ImageNet experiments indicate that these spectral interventions can improve generation metrics, and that combining amplification and suppression yields the best overall FID among the tested variants.
Novelty
The paper's main novelty is treating the denoiser Jacobian spectrum as a comparative diagnostic for generative denoising models and linking larger principal eigenvalues to better generative performance across pre-trained models. It also proposes a perturbation-based Jacobian regularization scheme that can either strengthen or suppress parts of the spectrum during training, including an explicit mechanism to increase Jacobian response along residual-aligned directions.
Results
On pre-trained SiT models, Jacobian eigenvalue statistics follow the same ordering as generative quality, while denoising error alone is not similarly informative. On ImageNet, residual-based regularization increases principal eigenvalues and improves FID over the SiT-S baseline, moderate stochastic regularization improves FID by suppressing spurious directions, and combining both regularizers gives the best reported overall FID among the tested settings.
Key Points
- The denoiser Jacobian spectrum provides a model-internal analysis tool that distinguishes generative denoisers in ways that sample-quality metrics alone do not.
- Residual perturbation regularization is designed to increase expansion along task-relevant eigen-directions, whereas stochastic perturbations act more like a contraction penalty on broader or random directions.
- In ImageNet experiments, moderate amplification of principal directions and suppression of weak noisy directions appear complementary, with the combined regularizer achieving the strongest overall FID in the reported comparisons.