Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusion

• URL: http://arxiv.org/abs/2603.14645v1
• Date: Sun, 15 Mar 2026 22:47:43 GMT
• Title: Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusion
• Abstract summary: We show that pixel-space diffusion trained with an MSE objective is inherently biased toward learning low and mid frequencies.<n>Motivated by this result, we propose the emphSpectrum Matching Hypothesis.<n>We show that Spectrum Matching yields superior diffusion generation on CelebA and ImageNet datasets.
• Score: 20.731913778080344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we study the diffusability (learnability) of variational autoencoders (VAE) in latent diffusion. First, we show that pixel-space diffusion trained with an MSE objective is inherently biased toward learning low and mid spatial frequencies, and that the power-law power spectral density (PSD) of natural images makes this bias perceptually beneficial. Motivated by this result, we propose the \emph{Spectrum Matching Hypothesis}: latents with superior diffusability should (i) follow a flattened power-law PSD (\emph{Encoding Spectrum Matching}, ESM) and (ii) preserve frequency-to-frequency semantic correspondence through the decoder (\emph{Decoding Spectrum Matching}, DSM). In practice, we apply ESM by matching the PSD between images and latents, and DSM via shared spectral masking with frequency-aligned reconstruction. Importantly, Spectrum Matching provides a unified view that clarifies prior observations of over-noisy or over-smoothed latents, and interprets several recent methods as special cases (e.g., VA-VAE, EQ-VAE). Experiments suggest that Spectrum Matching yields superior diffusion generation on CelebA and ImageNet datasets, and outperforms prior approaches. Finally, we extend the spectral view to representation alignment (REPA): we show that the directional spectral energy of the target representation is crucial for REPA, and propose a DoG-based method to further improve the performance of REPA. Our code is available https://github.com/forever208/SpectrumMatching.

Related papers

• Revisiting Spectral Representations in Generative Diffusion Models [98.43552218868837]
  Diffusion models have shown remarkable performance on diverse generation tasks.<n>Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality.<n>We investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels.
  arXiv  Detail & Related papers  (2026-09-08T04:54:02Z)
• HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models [9.926121668688772]
  Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence.<n>We identify a previously overlooked cause: spectral response rigidity.<n>We propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation.
  arXiv  Detail & Related papers  (2026-08-03T12:15:33Z)
• Calibrated Harmonic Overlaid Implicit Neural Representations for Multi-Dimensional Data [4.417873435160498]
  Implicit neural representation (INR) has emerged as a powerful prior for multi-dimensional data.<n>We propose a novel Calibrated Harmonic Overlaid Implicit Representation (CHOIR)<n>Our method achieves superior performance over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-06-25T08:48:31Z)
• SpecPL: Disentangling Spectral Granularity for Prompt Learning [11.908475229249161]
  We introduce Disentangling Spectral Granularity for Prompt Learning (SpecPL), which approaches prompt learning from a novel spectral perspective.<n> Specifically, we leverage a frozen VAE to decompose visual signals into semantic low-frequency bands and granular high-frequency details.<n>Experiments on 11 benchmarks demonstrate competitive state-of-the-art performance, achieving a new performance ceiling of 81.51% harmonic-mean accuracy.
  arXiv  Detail & Related papers  (2026-05-06T05:13:40Z)
• The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding [82.53463660564933]
  semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders retain high-frequency information that conveys fine-grained detail.<n>We propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator.
  arXiv  Detail & Related papers  (2025-12-22T18:59:57Z)
• Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability [21.927814986478356]
  We present a statistical analysis of video VAE latent spaces and identify two properties essential for diffusion training.<n>We propose two lightweight, backbone-agnostic regularizers: Local Correlation Regularization and Latent Masked Reconstruction.<n> Experiments show that our Spectral-Structured VAE achieves a $3times$ speedup in text-to-video convergence and a 10% gain in video reward, outperforming strong open-source VAEs.
  arXiv  Detail & Related papers  (2025-12-05T03:20:02Z)
• Uncertainty Quantification in HSI Reconstruction using Physics-Aware Diffusion Priors and Optics-Encoded Measurements [12.90606332269204]
  We formulate hyperspectral image (HSI) reconstruction as a Bayesian inference problem.<n>We propose a framework, HSDiff, that utilizes an unconditionally trained, pixel-level diffusion prior and posterior diffusion sampling.<n>We demonstrate that HSDiff offers a complete, high-performance method for uncertainty-aware HSI reconstruction.
  arXiv  Detail & Related papers  (2025-11-23T14:43:51Z)
• Latent Dirichlet Transformer VAE for Hyperspectral Unmixing with Bundled Endmembers [1.9336815376402718]
  We propose the Latent Dirichlet Transformer Variational Autoencoder (LDVAE-T) for hyperspectral unmixing.<n>Our model combines the global context modeling capabilities of transformer architectures with physically meaningful constraints imposed by a Dirichlet prior in the latent space.<n>We evaluate our approach on three benchmark datasets, Samson, Jasper Ridge, and HYDICE Urban.
  arXiv  Detail & Related papers  (2025-11-21T20:15:37Z)
• MARS-Sep: Multimodal-Aligned Reinforced Sound Separation [72.85468563236005]
  MARS-Sep is a reinforcement learning framework for sound separation.<n>It learns a factorized Beta mask policy that is optimized by a clipped trust-region surrogate.<n>Experiments on multiple benchmarks demonstrate consistent gains in Text-, Audio-, and Image-Queried separation.
  arXiv  Detail & Related papers  (2025-10-12T09:05:28Z)
• SpectrumFM: Redefining Spectrum Cognition via Foundation Modeling [65.65474629224558]
  We propose a spectrum foundation model, termed SpectrumFM, which provides a new paradigm for spectrum cognition.<n>An innovative spectrum encoder that exploits the convolutional neural networks is proposed to effectively capture both fine-grained local signal structures and high-level global dependencies in the spectrum data.<n>Two novel self-supervised learning tasks, namely masked reconstruction and next-slot signal prediction, are developed for pre-training SpectrumFM, enabling the model to learn rich and transferable representations.
  arXiv  Detail & Related papers  (2025-08-02T14:40:50Z)
• Frequency Domain-Based Diffusion Model for Unpaired Image Dehazing [92.61216319417208]
  We propose a novel frequency domain-based diffusion model, named ours, for fully exploiting the beneficial knowledge in unpaired clear data.<n>Inspired by the strong generative ability shown by Diffusion Models (DMs), we tackle the dehazing task from the perspective of frequency domain reconstruction.
  arXiv  Detail & Related papers  (2025-07-02T01:22:46Z)
• Spectrum Translation for Refinement of Image Generation (STIG) Based on Contrastive Learning and Spectral Filter Profile [15.5188527312094]
  We propose a framework to mitigate the disparity in frequency domain of the generated images. This is realized by spectrum translation for the refinement of image generation (STIG) based on contrastive learning. We evaluate our framework across eight fake image datasets and various cutting-edge models to demonstrate the effectiveness of STIG.
  arXiv  Detail & Related papers  (2024-03-08T06:39:24Z)
• Diffusion Probabilistic Model Made Slim [128.2227518929644]
  We introduce a customized design for slim diffusion probabilistic models (DPM) for light-weight image synthesis. We achieve 8-18x computational complexity reduction as compared to the latent diffusion models on a series of conditional and unconditional image generation tasks.
  arXiv  Detail & Related papers  (2022-11-27T16:27:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.