Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization

• URL: http://arxiv.org/abs/2607.00817v1
• Date: Wed, 01 Jul 2026 11:41:40 GMT
• Title: Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization
• Abstract summary: Text Embedding Steering (TES) is a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process.<n>We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters.
• Score: 15.96706698967133
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image diffusion models achieve impressive visual quality, yet demographic bias remains a challenge, as neutral prompts consistently produce stereotypical representations across gender and race. Existing approaches remain limited by costly retraining or by inference-time interventions that often degrade image quality and semantic alignment. We propose Text Embedding Steering (TES), a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process. We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters. Extensive experiments on Stable Diffusion demonstrate that TES outperforms existing training-free baselines in fairness while maintaining competitive image quality. These results highlight that inference-time text embedding optimization is a practical and scalable solution for fairness-aware generation in diffusion models.

Related papers

• Learning Late, Guiding Early: Timestep-Decoupled Semantic Guidance for Fair Face Generation [2.2993779624428208]
  We introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising.<n>SBP learns linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent.<n>Experiments on CelebA-HQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race.
  arXiv  Detail & Related papers  (2026-08-26T14:36:45Z)
• DebFilter: Eradicating Biases Stashed in Value [4.060731229044571]
  We propose DebFilter to mitigate social and semantic biases in text-to-image models.<n>We apply a bias-correction strategy that adjusts the value components within cross-attention.<n>Our results demonstrate that this method effectively reconfigures social biases in generated images.
  arXiv  Detail & Related papers  (2026-05-27T08:49:41Z)
• A Systematic Post-Train Framework for Video Generation [76.26555417456773]
  Large-scale video diffusion models have demonstrated impressive capabilities in generating high-resolution and semantically rich content.<n>A significant gap remains between their pretraining performance and real-world deployment requirements due to critical issues such as prompt sensitivity, temporal inconsistency, and prohibitive inference costs.<n>We propose a comprehensive post-training framework that systematically aligns pretrained models with user intentions through four synergistic stages.
  arXiv  Detail & Related papers  (2026-04-28T09:34:51Z)
• Your Pre-trained Diffusion Model Secretly Knows Restoration [55.7186754179308]
  We show that pre-trained diffusion models inherently possess restoration behavior, which can be unlocked by directly learning prompt embeddings.<n>We introduce our lightweight learned prompts on the pre-trained WAN video model and FLUX image models, converting them into high-performing restoration models.
  arXiv  Detail & Related papers  (2026-04-06T17:59:04Z)
• Reflective Flow Sampling Enhancement [51.824959905209774]
  We propose Reflective Flow Sampling (RF-Sampling) as a theoretically-grounded and training-free inference enhancement framework for flow models.<n>RF-Sampling consistently improves both generation quality and prompt alignment.<n>It is also the first inference enhancement method that can exhibit test-time scaling ability to some extent on FLUX.
  arXiv  Detail & Related papers  (2026-03-06T11:17:37Z)
• BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models [7.174865411448373]
  We propose a bilateral prompt optimization framework (BiPrompt) that simultaneously mitigates non-causal feature reliance in both modalities during test-time adaptation.<n>On the visual side, it employs structured attention-guided erasure to suppress background activations and enforce prediction consistency between causal and spurious regions.<n>On the textual side, it introduces balanced prompt normalization, a learnable re-centering mechanism that aligns class embeddings toward an isotropic semantic space.
  arXiv  Detail & Related papers  (2026-01-05T14:22:20Z)
• FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models [10.857020427374506]
  We introduce FairImagen, a post-hoc debiasing framework that operates on prompt embeddings to mitigate societal biases.<n>Our framework outperforms existing post-hoc methods and offers a simple, scalable, and model-agnostic solution for equitable text-to-image generation.
  arXiv  Detail & Related papers  (2025-10-24T11:47:15Z)
• Solving Inverse Problems with FLAIR [68.87167940623318]
  We present FLAIR, a training-free variational framework that leverages flow-based generative models as prior for inverse problems.<n>Results on standard imaging benchmarks demonstrate that FLAIR consistently outperforms existing diffusion- and flow-based methods in terms of reconstruction quality and sample diversity.
  arXiv  Detail & Related papers  (2025-06-03T09:29:47Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Text Diffusion with Reinforced Conditioning [92.17397504834825]
  This paper thoroughly analyzes text diffusion models and uncovers two significant limitations: degradation of self-conditioning during training and misalignment between training and sampling. Motivated by our findings, we propose a novel Text Diffusion model called TREC, which mitigates the degradation with Reinforced Conditioning and the misalignment by Time-Aware Variance Scaling.
  arXiv  Detail & Related papers  (2024-02-19T09:24:02Z)
• Exploiting Diffusion Prior for Real-World Image Super-Resolution [75.5898357277047]
  We present a novel approach to leverage prior knowledge encapsulated in pre-trained text-to-image diffusion models for blind super-resolution. By employing our time-aware encoder, we can achieve promising restoration results without altering the pre-trained synthesis model.
  arXiv  Detail & Related papers  (2023-05-11T17:55:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.