Revisiting Spectral Representations in Generative Diffusion Models
Abstract Overview
This paper revisits diffusion modeling through the lens of perturbation kernels and argues that diffusion training is closely connected to self-supervised spectral representation learning. Building on this view, the authors introduce a self-supervised spectral representation alignment method that regularizes diffusion model representations toward time-dependent spectral embeddings derived from the same perturbation process. The paper also provides a geometric interpretation in terms of preserving evolving local manifold structure during diffusion, and shows that the spectral objective is equivalent in gradient form to diffusion score distillation in representation space. Experiments on synthetic 2D data, image generation, and 3D point cloud generation indicate consistent improvements over matched diffusion baselines, especially in settings without external pretrained encoders.
Novelty
The main novelty is a unified perturbation-kernel framework that explicitly bridges diffusion models and spectral self-supervised learning, rather than treating representation alignment as an empirical add-on. The proposed alignment is fully self-supervised and avoids dependence on external teacher encoders, while the paper further demonstrates a theoretical equivalence between its spectral alignment objective and diffusion score distillation in representation space.
Results
Across image benchmarks, the method improves FID over corresponding baselines on all reported datasets, including ImageNet-64 (9.441 to 7.994), ImageNet-256 latent diffusion (2.508 to 2.298), CIFAR-10 (11.588 to 8.742), CelebA (28.806 to 25.678), and FFHQ (13.766 to 13.074). On 3D point cloud generation, it accelerates convergence and achieves stronger metrics; for instance, on ShapeNet Chairs at 5K iterations, 1-NNA drops from 0.850/0.875 to 0.583/0.627 and COV increases from 0.295/0.221 to 0.488/0.493 under CD/EMD. Synthetic 2D evaluations also demonstrate cleaner, more geometrically faithful samples with fewer outliers.
Key Points
- The method aligns diffusion representations with time-dependent spectral embeddings induced by the same diffusion perturbation kernel, providing a self-supervised alternative to encoder-based alignment methods such as REPA.
- The paper argues that spectral embeddings encode the evolving local geometry of the data manifold during diffusion and uses this as the motivation for adding spectral regularization to diffusion training.
- Empirically, the regularizer improves generation quality across images and 3D point clouds and is particularly relevant in domains where strong pretrained encoders are unavailable, although the paper notes added training overhead and weaker performance than REPA on large-scale latent ImageNet-256 generation.