Spectral Guidance for Flexible and Efficient Control of Diffusion Models

• URL: http://arxiv.org/abs/2605.28900v1
• Date: Wed, 27 May 2026 15:11:09 GMT
• Title: Spectral Guidance for Flexible and Efficient Control of Diffusion Models
• Abstract summary: We introduce Spectral Guidance, a framework for controlling diffusion models by leveraging the intrinsic geometry of the generative process.<n>We characterize them as the singular functions of a conditional expectation operator and show that they can be learned via a self-supervised objective.
• Score: 25.912824880444262
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Spectral Guidance, a framework for controlling diffusion models by leveraging the intrinsic geometry of the generative process. As data is progressively corrupted by noise, only a small number of features remain informative for control. We characterize them as the singular functions of a conditional expectation operator and show that they can be learned via a self-supervised objective. Once recovered, this basis enables the projection of arbitrary guidance signals, such as labels, CLIP embeddings, or masks, directly onto the sampling trajectory. This approach allows for stable, high-fidelity control without retraining or denoiser backpropagation during sampling. Empirically, we improve conditional accuracy on CIFAR-10 by 37 percentage points over the strongest training-free baseline while offering $4\times$ faster sampling. Moreover, the same representations that support label and CLIP guidance also enable spatial control, such as mask-based guidance, without auxiliary models. Finally, our framework reveals a phase transition in the generative process, pinpointing the optimal time window for effective guidance.

Related papers

• PathSelect: Sequential Token Selection for Whole Slide Pathology [0.6362628914555645]
  Whole-Slide Images (WSIs) present a fundamental computational bottleneck for vision-language models (VLMs) due to extreme sequence lengths.<n>We reformulate WSI token pruning as a sequential selection process, enabling the model to autonomously learn an optimal routing strategy.
  arXiv  Detail & Related papers  (2026-07-26T12:36:16Z)
• From Open Loop to Closed Loop: A Test-Time Iterative Optimization Framework for Reference-Consistent Image Generation [108.02425846269234]
  controllable image generation has made significant strides by incorporating visual reference conditions.<n>Existing methods predominantly operate as open-loop systems.<n>We propose a novel test-time iterative optimization framework that reformulates reference-consistent generation as a closed-loop dynamic tracking problem.
  arXiv  Detail & Related papers  (2026-07-06T05:36:30Z)
• Exploring the Limits of End-to-End Feature-Affinity Propagation for Single-Point Supervised Infrared Small Target Detection [6.7126559398433665]
  Single-point supervised infrared small target detection (IRSTD) drastically reduces dense annotation costs.<n>Current state-of-the-art (SOTA) methods achieve high precision by recovering mask supervision through explicit, offline pseudo-label construction.<n>We study a minimalist alternative: generating point-to-mask supervision online through in-batch, point-anchored feature-affinity propagation.
  arXiv  Detail & Related papers  (2026-05-01T15:08:46Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• General and Efficient Steering of Unconditional Diffusion [25.225845714398364]
  We present a recipe for efficiently steering unconditional diffusion.<n>without gradient guidance during inference.<n>Our approach is built on two observations about diffusion model structure.<n>Experiments on CIFAR-10, ImageNet, and CelebA demonstrate improved accuracy/quality over-based gradient guidance.
  arXiv  Detail & Related papers  (2026-02-11T21:58:26Z)
• Controllable and explainable personality sliders for LLMs at inference time [1.3688381983244782]
  We propose a modular framework for continuous, multi-dimensional personality control.<n>Our key innovation is Sequential Adaptive Steering (SAS), a method that transforms steering vectors by training subsequent probes on the residual stream shifted by prior interventions.<n>We validate our framework on the Big Five personality traits, demonstrating that it outperforms naive baselines in both goal adherence and coherence.
  arXiv  Detail & Related papers  (2026-02-10T08:16:59Z)
• Learnable Chernoff Baselines for Inference-Time Alignment [64.81256817158851]
  We introduce Learnable Chernoff Baselines as a method for efficiently and approximately sampling from exponentially tilted kernels.<n>We establish total-variation guarantees to the ideal aligned model, and demonstrate in both continuous and discrete diffusion settings that LCB sampling closely matches ideal rejection sampling.
  arXiv  Detail & Related papers  (2026-02-08T00:09:40Z)
• RAAG: Ratio Aware Adaptive Guidance [9.525432706814675]
  Flow-based generative models have achieved remarkable progress.<n>Applying a strong, fixed guidance scale throughout inference is poorly suited for the rapid, few-step sampling required by modern applications.<n>We propose a simple, theoretically grounded, adaptive guidance schedule that automatically dampens the guidance scale at early steps based on the evolving ratio.
  arXiv  Detail & Related papers  (2025-08-05T13:41:05Z)
• Semi-Supervised Class-Agnostic Motion Prediction with Pseudo Label Regeneration and BEVMix [59.55173022987071]
  We study the potential of semi-supervised learning for class-agnostic motion prediction. Our framework adopts a consistency-based self-training paradigm, enabling the model to learn from unlabeled data. Our method exhibits comparable performance to weakly and some fully supervised methods.
  arXiv  Detail & Related papers  (2023-12-13T09:32:50Z)
• Readout Guidance: Learning Control from Diffusion Features [96.22155562120231]
  We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity.
  arXiv  Detail & Related papers  (2023-12-04T18:59:32Z)
• Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimization [87.21285093582446]
  Diffusion Generative Flow Samplers (DGFS) is a sampling-based framework where the learning process can be tractably broken down into short partial trajectory segments. Our method takes inspiration from the theory developed for generative flow networks (GFlowNets)
  arXiv  Detail & Related papers  (2023-10-04T09:39:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.