FairGen: Controlling Sensitive Attributes for Fair Generations in Diffusion Models via Adaptive Latent Guidance

• URL: http://arxiv.org/abs/2503.01872v1
• Date: Tue, 25 Feb 2025 23:47:22 GMT
• Title: FairGen: Controlling Sensitive Attributes for Fair Generations in Diffusion Models via Adaptive Latent Guidance
• Authors: Mintong Kang, Vinayshekhar Bannihatti Kumar, Shamik Roy, Abhishek Kumar, Sopan Khosla, Balakrishnan Murali Narayanaswamy, Rashmi Gangadharaiah,
• Abstract summary: Text-to-image diffusion models often exhibit biases toward specific demographic groups.<n>In this paper, we tackle the challenge of mitigating generation bias towards any target attribute value.<n>We propose FairGen, an adaptive latent guidance mechanism which controls the generation distribution during inference.
• Score: 19.65226469682089
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image diffusion models often exhibit biases toward specific demographic groups, such as generating more males than females when prompted to generate images of engineers, raising ethical concerns and limiting their adoption. In this paper, we tackle the challenge of mitigating generation bias towards any target attribute value (e.g., "male" for "gender") in diffusion models while preserving generation quality. We propose FairGen, an adaptive latent guidance mechanism which controls the generation distribution during inference. In FairGen, a latent guidance module dynamically adjusts the diffusion process to enforce specific attributes, while a memory module tracks the generation statistics and steers latent guidance to align with the targeted fair distribution of the attribute values. Further, given the limitations of existing datasets in comprehensively assessing bias in diffusion models, we introduce a holistic bias evaluation benchmark HBE, covering diverse domains and incorporating complex prompts across various applications. Extensive evaluations on HBE and Stable Bias datasets demonstrate that FairGen outperforms existing bias mitigation approaches, achieving substantial bias reduction (e.g., 68.5% gender bias reduction on Stable Diffusion 2). Ablation studies highlight FairGen's ability to flexibly and precisely control generation distribution at any user-specified granularity, ensuring adaptive and targeted bias mitigation.

Related papers

• DebiasDiff: Debiasing Text-to-image Diffusion Models with Self-discovering Latent Attribute Directions [16.748044041907367]
  DebiasDiff is a plug-and-play method that learns attribute latent directions in a self-discovering manner.<n>Our method enables debiasing multiple attributes in DMs simultaneously, while remaining lightweight and easily integrable with other DMs.
  arXiv  Detail & Related papers  (2024-12-25T07:30:20Z)
• FADE: Towards Fairness-aware Generation for Domain Generalization via Classifier-Guided Score-based Diffusion Models [14.962882545521675]
  Fairness-aware domain generalization (FairDG) has emerged as a critical challenge for deploying trustworthy AI systems. Traditional methods for addressing fairness have failed in domain generalization due to their lack of consideration for distribution shifts. We propose Fairness-aware Score-Guided Diffusion Models (FADE) as a novel approach to effectively address the FairDG issue.
  arXiv  Detail & Related papers  (2024-06-13T17:36:05Z)
• Distributionally Generative Augmentation for Fair Facial Attribute Classification [69.97710556164698]
  Facial Attribute Classification (FAC) holds substantial promise in widespread applications. FAC models trained by traditional methodologies can be unfair by exhibiting accuracy inconsistencies across varied data subpopulations. This work proposes a novel, generation-based two-stage framework to train a fair FAC model on biased data without additional annotation.
  arXiv  Detail & Related papers  (2024-03-11T10:50:53Z)
• Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models [59.331993845831946]
  Diffusion models benefit from instillation of task-specific information into the score function to steer the sample generation towards desired properties. This paper provides the first theoretical study towards understanding the influence of guidance on diffusion models in the context of Gaussian mixture models.
  arXiv  Detail & Related papers  (2024-03-03T23:15:48Z)
• Balancing Act: Distribution-Guided Debiasing in Diffusion Models [31.38505986239798]
  Diffusion Models (DMs) have emerged as powerful generative models with unprecedented image generation capability. DMs reflect the biases present in the training datasets. We present a method for debiasing DMs without relying on additional data or model retraining.
  arXiv  Detail & Related papers  (2024-02-28T09:53:17Z)
• Gaussian Harmony: Attaining Fairness in Diffusion-based Face Generation Models [31.688873613213392]
  Diffusion models amplify the bias in the generation process, leading to an imbalance in distribution of sensitive attributes such as age, gender and race. We mitigate the bias by localizing the means of the facial attributes in the latent space of the diffusion model using Gaussian mixture models (GMM) Our results demonstrate that our approach leads to a more fair data generation in terms of representational fairness while preserving the quality of generated samples.
  arXiv  Detail & Related papers  (2023-12-21T20:06:15Z)
• Causality and Independence Enhancement for Biased Node Classification [56.38828085943763]
  We propose a novel Causality and Independence Enhancement (CIE) framework, applicable to various graph neural networks (GNNs) Our approach estimates causal and spurious features at the node representation level and mitigates the influence of spurious correlations. Our approach CIE not only significantly enhances the performance of GNNs but outperforms state-of-the-art debiased node classification methods.
  arXiv  Detail & Related papers  (2023-10-14T13:56:24Z)
• Toward Fair Facial Expression Recognition with Improved Distribution Alignment [19.442685015494316]
  We present a novel approach to mitigate bias in facial expression recognition (FER) models. Our method aims to reduce sensitive attribute information such as gender, age, or race, in the embeddings produced by FER models. For the first time, we analyze the notion of attractiveness as an important sensitive attribute in FER models and demonstrate that FER models can indeed exhibit biases towards more attractive faces.
  arXiv  Detail & Related papers  (2023-06-11T14:59:20Z)
• Analyzing Bias in Diffusion-based Face Generation Models [75.80072686374564]
  Diffusion models are increasingly popular in synthetic data generation and image editing applications. We investigate the presence of bias in diffusion-based face generation models with respect to attributes such as gender, race, and age. We examine how dataset size affects the attribute composition and perceptual quality of both diffusion and Generative Adversarial Network (GAN) based face generation models.
  arXiv  Detail & Related papers  (2023-05-10T18:22:31Z)
• Self-Conditioned Generative Adversarial Networks for Image Editing [61.50205580051405]
  Generative Adversarial Networks (GANs) are susceptible to bias, learned from either the unbalanced data, or through mode collapse. We argue that this bias is responsible not only for fairness concerns, but that it plays a key role in the collapse of latent-traversal editing methods when deviating away from the distribution's core.
  arXiv  Detail & Related papers  (2022-02-08T18:08:24Z)
• Balancing Biases and Preserving Privacy on Balanced Faces in the Wild [50.915684171879036]
  There are demographic biases present in current facial recognition (FR) models. We introduce our Balanced Faces in the Wild dataset to measure these biases across different ethnic and gender subgroups. We find that relying on a single score threshold to differentiate between genuine and imposters sample pairs leads to suboptimal results. We propose a novel domain adaptation learning scheme that uses facial features extracted from state-of-the-art neural networks.
  arXiv  Detail & Related papers  (2021-03-16T15:05:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.