Learning Late, Guiding Early: Timestep-Decoupled Semantic Guidance for Fair Face Generation

• URL: http://arxiv.org/abs/2608.25862v2
• Date: Thu, 27 Aug 2026 09:01:59 GMT
• Title: Learning Late, Guiding Early: Timestep-Decoupled Semantic Guidance for Fair Face Generation
• Abstract summary: We introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising.<n>SBP learns linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent.<n>Experiments on CelebA-HQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race.
• Score: 2.2993779624428208
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Demographic imbalance in synthetic face generation can propagate to downstream face recognition systems, making fairness an important consideration when diffusion models are used for data generation. Existing fairness-aware generation approaches often require model retraining, architectural modifications, or repeated guidance throughout the reverse diffusion process. In this work, we introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising. Our approach is motivated by the observation that latent representations at different diffusion timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention. SBP leverages this timestep decoupling by learning linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent, allowing the remainder of the reverse denoising process to proceed unchanged. The method requires neither retraining nor fine-tuning of the underlying Latent Diffusion Model and operates without external balanced datasets. Experiments on CelebA-HQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race, while maintaining perceptual image quality across demographic groups. Owing to its one-shot inference strategy and model-agnostic design, SBP introduces only a small computational overhead and can be readily integrated with existing pre-trained latent diffusion models.

Related papers

• Probing Diffusion Denoising Dynamics for Contrastive Representation Learning [48.679227622825806]
  We study how can the denoising dynamics of a pretrained diffusion model be adapted to support discriminative representation learning.<n>We present D$3$CL as an investigation of this question.<n>We show that noise-level contrastive learning can structure denoising representations for discriminative tasks while maintaining generative performance.
  arXiv  Detail & Related papers  (2026-07-10T03:23:44Z)
• Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization [15.96706698967133]
  Text Embedding Steering (TES) is a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process.<n>We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters.
  arXiv  Detail & Related papers  (2026-07-01T11:41:40Z)
• Temporal Difference Learning for Diffusion Models [19.82560792699663]
  We introduce a temporal difference (TD) objective that penalizes inconsistency of the model's multi-step progress along the denoising path.<n>We propose a principled sample-based reweighting method that stabilizes training.
  arXiv  Detail & Related papers  (2026-06-13T01:34:04Z)
• Interpolating Discrete Diffusion Models with Controllable Resampling [49.771004594828405]
  Masked diffusion models suffer from irreversible errors due to early unmasking.<n>Uniform diffusion models, despite enabling self-correction, often yield low-quality samples.<n>We introduce Interpolating Discrete Diffusion Model, that improves diffusion by reducing dependence on intermediate latent states.
  arXiv  Detail & Related papers  (2026-04-19T07:55:57Z)
• Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Models [11.196851704643406]
  Diffusion language models (Diffusion-LMs) introduce an explicit temporal dimension into text generation.<n>We show that Diffusion-LMs, like diffusion models in image generation, exhibit a temporal division of labor.<n>We propose Time-Annealed Perturbation Sampling (TAPS), a training-free inference strategy that encourages semantic branching early in the diffusion process.<n>TAPS is compatible with both non-autoregressive and semi-autoregressive Diffusion backbones, demonstrated on LLaDA and TraDo in our paper, and consistently improves output diversity across creative writing and reasoning benchmarks without compromising generation quality.
  arXiv  Detail & Related papers  (2026-01-30T06:39:33Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production [0.0]
  We develop a hybrid approach that combines autoregressive and diffusion models for Sign Language Production (SLP)<n>To capture fine-grained body movements, we design a Multi-Scale Pose Representation module that separately extracts detailed features from distinct articulators.<n>We introduce a Confidence-Aware Causal Attention mechanism that utilizes joint-level confidence scores to dynamically guide the pose generation process.
  arXiv  Detail & Related papers  (2025-07-12T01:34:50Z)
• Exploring Representation-Aligned Latent Space for Better Generation [86.45670422239317]
  We introduce ReaLS, which integrates semantic priors to improve generation performance.<n>We show that fundamental DiT and SiT trained on ReaLS can achieve a 15% improvement in FID metric.<n>The enhanced semantic latent space enables more perceptual downstream tasks, such as segmentation and depth estimation.
  arXiv  Detail & Related papers  (2025-02-01T07:42:12Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors [56.82596340418697]
  We propose a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.
  arXiv  Detail & Related papers  (2024-01-29T10:36:57Z)
• One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls [77.42510898755037]
  One More Step (OMS) is a compact network that incorporates an additional simple yet effective step during inference. OMS elevates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model parameters. Once trained, various pre-trained diffusion models with the same latent domain can share the same OMS module.
  arXiv  Detail & Related papers  (2023-11-27T12:02:42Z)
• Exploring Compositional Visual Generation with Latent Classifier Guidance [19.48538300223431]
  We train latent diffusion models and auxiliary latent classifiers to facilitate non-linear navigation of latent representation generation. We show that such conditional generation achieved by latent classifier guidance provably maximizes a lower bound of the conditional log probability during training. We show that this paradigm based on latent classifier guidance is agnostic to pre-trained generative models, and present competitive results for both image generation and sequential manipulation of real and synthetic images.
  arXiv  Detail & Related papers  (2023-04-25T03:02:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.