Towards More General Control of Diffusion Models Using Jeffrey Guidance

• URL: http://arxiv.org/abs/2606.13240v1
• Date: Thu, 11 Jun 2026 11:53:53 GMT
• Title: Towards More General Control of Diffusion Models Using Jeffrey Guidance
• Abstract summary: Jeffrey guidance is a principled framework that extends diffusion-model control to applications beyond what standard guidance can express.<n>We first demonstrate Jeffrey guidance by targeting a prescribed embedding distribution.<n>With Inception embeddings as the target, this leads to substantial reductions in FID on both CIFAR-10 and FFHQ.
• Score: 21.06046163920049
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A key strength of diffusion models lies in their flexibility, since their outputs can be controlled at sampling time through guidance. However, beyond simple cases such as conditional sampling, the target distribution is often left implicit, defined only through a sampling rule or a heuristic energy function. To address this, we propose Jeffrey guidance, a principled framework that extends diffusion-model control to applications beyond what standard guidance can express. It leverages Jeffrey's rule of conditioning to update marginal distributions towards a prescribed target, preserving the conditional structure and minimally perturbing the joint distribution. We first demonstrate Jeffrey guidance by targeting a prescribed embedding distribution. With Inception embeddings as the target, this leads to substantial reductions in FID on both CIFAR-10 and FFHQ. We further apply Jeffrey guidance to fairness on CelebA-HQ, updating an unconditional diffusion model to enforce independence between attributes.

Related papers

• A Mean-Field Framework for Inference-Time Distributional Control of Diffusion Models [3.2299643462065286]
  We formulate inference-time distributional control as targeting a tilted measure under a mean-field framework.<n>We derive a weighted interacting particle scheme to target it in a principled manner.<n> Empirically, we verify that the procedure correctly targets the prescribed distribution in tractable low-dimensional settings.
  arXiv  Detail & Related papers  (2026-08-09T15:41:01Z)
• Inference-Time Attribute Distribution Alignment for Unconditional Diffusion [54.28942858023809]
  We formalize the inference-time attribute distributional alignment problem for pretrained unconditional diffusion models.<n>To address this, we cast inference-time attribute distributional alignment as an optimal control problem over the reverse diffusion process.<n>We solve for the perturbations using an optimal-control-based algorithm to optimize a differentiable distribution-matching objective.
  arXiv  Detail & Related papers  (2026-05-08T09:02:47Z)
• Guiding a Diffusion Model by Swapping Its Tokens [16.588428780117752]
  We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation.<n>The key idea is to generate a perturbed prediction via simple token swap operations.<n>Our approach selectively exchanges and recomposes token latents, allowing finer control over perturbation.
  arXiv  Detail & Related papers  (2026-04-09T09:54:49Z)
• C$^2$FG: Control Classifier-Free Guidance via Score Discrepancy Analysis [25.481226578749418]
  We provide a rigorous theoretical analysis of the inherent dynamics of free guidance.<n>We show that C$2$FG is effective and broadly applicable across diverse generative tasks.
  arXiv  Detail & Related papers  (2026-03-09T09:37:17Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Feedback Guidance of Diffusion Models [14.162420300295365]
  Interval-Free Guidance (CFG) has become standard for improving sample fidelity in conditional diffusion models.<n>We propose FeedBack Guidance (FBG), which uses a state-dependent coefficient to self-regulate guidance amounts based on need.
  arXiv  Detail & Related papers  (2025-06-06T13:46:32Z)
• Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models [57.20761595019967]
  We present Normalized Attention Guidance (NAG), an efficient, training-free mechanism that applies extrapolation in attention space with L1-based normalization and refinement.<n>NAG restores effective negative guidance where CFG collapses while maintaining fidelity.<n>NAG generalizes across architectures (UNet, DiT), sampling regimes (few-step, multi-step), and modalities (image, video)
  arXiv  Detail & Related papers  (2025-05-27T13:30:46Z)
• Rectified Diffusion Guidance for Conditional Generation [94.83538269086613]
  We revisit the theory behind CFG and rigorously confirm that the improper combination coefficients (textiti.e.) brings about expectation shift the generative distribution.<n>We show that our approach enjoys a textbftextitform solution given the strength.<n> Empirical evidence on real-world data demonstrate the compatibility of our design with existing state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2024-10-24T13:41:32Z)
• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional dependencies for general score-mismatched diffusion samplers.<n>We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions.<n>This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• Manifold Preserving Guided Diffusion [121.97907811212123]
  Conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. We propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework.
  arXiv  Detail & Related papers  (2023-11-28T02:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.