SteeringDiffusion: A Bottlenecked Activation Control Interface for Diffusion Models

• URL: http://arxiv.org/abs/2605.01653v1
• Date: Sun, 03 May 2026 00:29:10 GMT
• Title: SteeringDiffusion: A Bottlenecked Activation Control Interface for Diffusion Models
• Abstract summary: SteeringDiffusion is a bottlenecked activation-level control interface for diffusion models.<n>A zero-awared design guarantees exact equivalence to the base model at zero scale.<n>Across experiments on Stable Diffusion1.5 and SDXL, we show that SteeringDiffusion produces smooth and monotonic content-style trade-offs.
• Score: 1.9336815376402718
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce SteeringDiffusion, a bottlenecked activation-level control interface for diffusion models that exposes a smooth, monotonic, and runtime-adjustable control surface over the content--style trade-off. Our method keeps the U-Net backbone frozen and learns a small, prompt-conditioned latent code projected to FiLM/AdaGN-style modulation parameters. A zero-initialized design guarantees exact equivalence to the base model at zero scale, while timestep-aware gating restricts modulation to later denoising stages. A single scalar at inference continuously traverses the control surface without retraining. Across experiments on Stable Diffusion~1.5 and SDXL covering multiple artistic styles, we show that SteeringDiffusion produces smooth and monotonic content--style trade-offs. Under matched parameter budgets, it outperforms LoRA in controllability and stability, while ControlNet and rank-1 adapters do not expose a comparable control surface. We further introduce an inversion-stability diagnostic based on DDIM inversion, used as a post-hoc trajectory probe, which reveals strong correlations with intervention magnitude. These results position \emph{Steering Bottlenecked Explicit Control (S-BEC)} as a practical, general-purpose control interface for frozen diffusion backbones.

Related papers

• Closed-Loop Neural Activation Control in Vision-Language-Action Models [19.22723477431829]
  We propose a closed-loop framework that replaces static intervention strength with adaptive, time-varying control signals.<n>We instantiate this framework with both PID and reinforcement learning based controllers.<n> Experiments show that residual-STEER achieves more stable concept regulation and a better steering-task success trade-off than fixed-coefficient baselines.
  arXiv  Detail & Related papers  (2026-05-29T18:59:53Z)
• Controllable Text-to-Motion Generation via Modular Body-Part Phase Control [25.2003430574182]
  Text-to-motion (T2M) generation is becoming a practical tool for animation and interactive avatars.<n>We propose Modular Body-Part Phase Control, a plug-and-play framework enabling structured, localized editing.<n> Experiments on both diffusion- and flow-based models demonstrate that our approach provides predictable and fine-grained control over motion magnitude, speed, and timing.
  arXiv  Detail & Related papers  (2026-03-20T09:37:16Z)
• Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer [55.9892973179428]
  MoTok is a discrete motion tokenizer that decouples semantic abstraction from fine-grained reconstruction.<n>On HumanML3D, our method significantly improves controllability and fidelity over MaskControl while using only one-sixth of the tokens.
  arXiv  Detail & Related papers  (2026-03-19T17:59:51Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• CFG-Ctrl: Control-Based Classifier-Free Diffusion Guidance [31.552164852288325]
  We introduce Sliding Mode Control CFG (SMC-CFG), which enforces the generative flow toward a rapidly convergent sliding manifold.<n>SMC-CFG outperforms standard CFG in semantic alignment and robustness across a wide range of guidance scales.
  arXiv  Detail & Related papers  (2026-03-03T18:59:48Z)
• ACD: Direct Conditional Control for Video Diffusion Models via Attention Supervision [62.41380823195191]
  We propose Attention-Conditional Diffusion, a framework for direct conditional control in video diffusion models via attention supervision.<n>ACD achieves better controllability by aligning the model's attention maps with external control signals.<n>Experiments on benchmark video generation datasets demonstrate that ACD delivers superior alignment with conditioning inputs.
  arXiv  Detail & Related papers  (2025-12-24T16:24:18Z)
• ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attention [86.93601565563954]
  ScaleWeaver is a framework designed to achieve high-fidelity, controllable generation upon advanced visual autoregressive( VAR) models.<n>The proposed Reference Attention module discards the unnecessary attention from image$rightarrow$condition, reducing computational cost.<n>Experiments show that ScaleWeaver delivers high-quality generation and precise control while attaining superior efficiency over diffusion-based methods.
  arXiv  Detail & Related papers  (2025-10-16T17:00:59Z)
• DiffusionNFT: Online Diffusion Reinforcement with Forward Process [99.94852379720153]
  Diffusion Negative-aware FineTuning (DiffusionNFT) is a new online RL paradigm that optimize diffusion models directly on the forward process via flow matching.<n>DiffusionNFT is up to $25times$ more efficient than FlowGRPO in head-to-head comparisons, while being CFG-free.
  arXiv  Detail & Related papers  (2025-09-19T16:09:33Z)
• NanoControl: A Lightweight Framework for Precise and Efficient Control in Diffusion Transformer [14.644014499085943]
  NanoControl employs Flux as the backbone network for controllable text-to-image generation.<n>Our model achieves state-of-the-art controllable text-to-image generation performance.<n>It incurs only a 0.024% increase in parameter count and a 0.029% increase in GFLOPs, thus enabling highly efficient controllable generation.
  arXiv  Detail & Related papers  (2025-08-14T07:54:44Z)
• Adding Additional Control to One-Step Diffusion with Joint Distribution Matching [58.37264951734603]
  JDM is a novel approach that minimizes the reverse KL divergence between image-condition joint distributions.<n>By deriving a tractable upper bound, JDM decouples fidelity learning from condition learning.<n>This asymmetric distillation scheme enables our one-step student to handle controls unknown to the teacher model.
  arXiv  Detail & Related papers  (2025-03-09T15:06:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.