General and Efficient Steering of Unconditional Diffusion

• URL: http://arxiv.org/abs/2602.11395v1
• Date: Wed, 11 Feb 2026 21:58:26 GMT
• Title: General and Efficient Steering of Unconditional Diffusion
• Authors: Qingsong Wang, Mikhail Belkin, Yusu Wang,
• Abstract summary: We present a recipe for efficiently steering unconditional diffusion.<n>without gradient guidance during inference.<n>Our approach is built on two observations about diffusion model structure.<n>Experiments on CIFAR-10, ImageNet, and CelebA demonstrate improved accuracy/quality over-based gradient guidance.
• Score: 25.225845714398364
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Guiding unconditional diffusion models typically requires either retraining with conditional inputs or per-step gradient computations (e.g., classifier-based guidance), both of which incur substantial computational overhead. We present a general recipe for efficiently steering unconditional diffusion {without gradient guidance during inference}, enabling fast controllable generation. Our approach is built on two observations about diffusion model structure: Noise Alignment: even in early, highly corrupted stages, coarse semantic steering is possible using a lightweight, offline-computed guidance signal, avoiding any per-step or per-sample gradients. Transferable concept vectors: a concept direction in activation space once learned transfers across both {timesteps} and {samples}; the same fixed steering vector learned near low noise level remains effective when injected at intermediate noise levels for every generation trajectory, providing refined conditional control with efficiency. Such concept directions can be efficiently and reliably identified via Recursive Feature Machine (RFM), a light-weight backpropagation-free feature learning method. Experiments on CIFAR-10, ImageNet, and CelebA demonstrate improved accuracy/quality over gradient-based guidance, while achieving significant inference speedups.

Related papers

• Path-Guided Flow Matching for Dataset Distillation [9.761850986508895]
  We propose the first flow matching-based framework for generative distillation, which enables fast deterministic synthesis by solving an ODE in a few steps.<n>We develop a continuous path-to-prototype guidance algorithm for ODE-consistent path control, which allows trajectories to reliably land on assigned prototypes.
  arXiv  Detail & Related papers  (2026-02-05T12:52:32Z)
• Spatial-Temporal Feedback Diffusion Guidance for Controlled Traffic Imputation [17.509468754018496]
  Imputing missing values in spatial-temporal traffic data is essential for intelligent transportation systems.<n>We propose FENCE, a spatial-temporal feedback diffusion guidance method designed to adaptively control guidance scales during imputation.<n>FENCE computes guidance scales at the cluster level by grouping nodes based on their attention scores.
  arXiv  Detail & Related papers  (2026-01-08T04:03:32Z)
• TAG:Tangential Amplifying Guidance for Hallucination-Resistant Diffusion Sampling [53.61290359948953]
  Tangential Amplifying Guidance (TAG) operates solely on trajectory signals without modifying the underlying diffusion model.<n>We formalize this guidance process by leveraging a first-order Taylor expansion.<n> TAG is a plug-and-play, architecture-agnostic module that improves diffusion sampling fidelity with minimal computational addition.
  arXiv  Detail & Related papers  (2025-10-06T06:53:29Z)
• Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching [36.348940136801296]
  A novel guidance framework for discrete data is proposed to address this problem.<n>We derive the exact transition rate for the desired distribution given a learned discrete flow matching model.<n>We demonstrate the effectiveness of our proposed guidance on energy-guided simulations and preference alignment on text-to-image generation and multimodal understanding tasks.
  arXiv  Detail & Related papers  (2025-09-26T05:51:31Z)
• Noise-Level Diffusion Guidance: Well Begun is Half Done [9.745109820010601]
  Noise Level Guidance (NLG) is a simple, efficient, and general noise-level optimization approach.<n>It refines initial noise by increasing the likelihood of its alignment with general guidance.<n>Our method establishes NLG as a practical and scalable enhancement to diffusion models.
  arXiv  Detail & Related papers  (2025-09-17T13:05:59Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• Diffusion-Sharpening: Fine-tuning Diffusion Models with Denoising Trajectory Sharpening [56.99266993852532]
  Diffusion-Sharpening is a fine-tuning approach that enhances downstream alignment by optimizing sampling trajectories.<n>Our method demonstrates superior training efficiency with faster convergence, and best inference efficiency without requiring additional NFEs.
  arXiv  Detail & Related papers  (2025-02-17T18:57:26Z)
• Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation [49.49868273653921]
  Diffusion models are promising for joint trajectory prediction and controllable generation in autonomous driving. We introduce Optimal Gaussian Diffusion (OGD) and Estimated Clean Manifold (ECM) Guidance. Our methodology streamlines the generative process, enabling practical applications with reduced computational overhead.
  arXiv  Detail & Related papers  (2024-08-01T17:59:59Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models [63.1637853118899]
  We propose the first unsupervised and learning-based method to identify interpretable directions in h-space of pre-trained diffusion models. We employ a shift control module that works on h-space of pre-trained diffusion models to manipulate a sample into a shifted version of itself. By jointly optimizing them, the model will spontaneously discover disentangled and interpretable directions.
  arXiv  Detail & Related papers  (2023-10-15T18:44:30Z)
• Observation-Guided Diffusion Probabilistic Models [41.749374023639156]
  We propose a novel diffusion-based image generation method called the observation-guided diffusion probabilistic model (OGDM) Our approach reestablishes the training objective by integrating the guidance of the observation process with the Markov chain. We demonstrate the effectiveness of our training algorithm using diverse inference techniques on strong diffusion model baselines.
  arXiv  Detail & Related papers  (2023-10-06T06:29:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.