Fair Generation without Unfair Distortions: Debiasing Text-to-Image Generation with Entanglement-Free Attention

• URL: http://arxiv.org/abs/2506.13298v2
• Date: Sun, 03 Aug 2025 05:30:11 GMT
• Title: Fair Generation without Unfair Distortions: Debiasing Text-to-Image Generation with Entanglement-Free Attention
• Authors: Jeonghoon Park, Juyoung Lee, Chaeyeon Chung, Jaeseong Lee, Jaegul Choo, Jindong Gu,
• Abstract summary: Entanglement-Free Attention (EFA) is a method that accurately incorporates target attributes while preserving non-target attributes during bias mitigation.<n>At inference time, EFA randomly samples a target attribute with equal probability and adjusts the cross-attention in selected layers to incorporate the sampled attribute.<n>Extensive experiments demonstrate that EFA outperforms existing methods in mitigating bias while preserving non-target attributes.
• Score: 42.277875137852234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in diffusion-based text-to-image (T2I) models have enabled the generation of high-quality and photorealistic images from text. However, they often exhibit societal biases related to gender, race, and socioeconomic status, thereby potentially reinforcing harmful stereotypes and shaping public perception in unintended ways. While existing bias mitigation methods demonstrate effectiveness, they often encounter attribute entanglement, where adjustments to attributes relevant to the bias (i.e., target attributes) unintentionally alter attributes unassociated with the bias (i.e., non-target attributes), causing undesirable distribution shifts. To address this challenge, we introduce Entanglement-Free Attention (EFA), a method that accurately incorporates target attributes (e.g., White, Black, and Asian) while preserving non-target attributes (e.g., background) during bias mitigation. At inference time, EFA randomly samples a target attribute with equal probability and adjusts the cross-attention in selected layers to incorporate the sampled attribute, achieving a fair distribution of target attributes. Extensive experiments demonstrate that EFA outperforms existing methods in mitigating bias while preserving non-target attributes, thereby maintaining the original model's output distribution and generative capacity.

Related papers

• Bias Analysis in Unconditional Image Generative Models [21.530188920526843]
  We train a set of unconditional image generative models and adopt a commonly used bias evaluation framework to study bias shift between training and generated distributions.<n>Our experiments reveal that the detected attribute shifts are small.<n>We find that the attribute shifts are sensitive to the attribute classifier used to label generated images in the evaluation framework, particularly when its decision boundaries fall in high-density regions.
  arXiv  Detail & Related papers  (2025-06-10T16:53:10Z)
• FairGen: Controlling Sensitive Attributes for Fair Generations in Diffusion Models via Adaptive Latent Guidance [19.65226469682089]
  Text-to-image diffusion models often exhibit biases toward specific demographic groups.<n>In this paper, we tackle the challenge of mitigating generation bias towards any target attribute value.<n>We propose FairGen, an adaptive latent guidance mechanism which controls the generation distribution during inference.
  arXiv  Detail & Related papers  (2025-02-25T23:47:22Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• AITTI: Learning Adaptive Inclusive Token for Text-to-Image Generation [53.65701943405546]
  We learn adaptive inclusive tokens to shift the attribute distribution of the final generative outputs. Our method requires neither explicit attribute specification nor prior knowledge of the bias distribution. Our method achieves comparable performance to models that require specific attributes or editing directions for generation.
  arXiv  Detail & Related papers  (2024-06-18T17:22:23Z)
• Distributionally Generative Augmentation for Fair Facial Attribute Classification [69.97710556164698]
  Facial Attribute Classification (FAC) holds substantial promise in widespread applications. FAC models trained by traditional methodologies can be unfair by exhibiting accuracy inconsistencies across varied data subpopulations. This work proposes a novel, generation-based two-stage framework to train a fair FAC model on biased data without additional annotation.
  arXiv  Detail & Related papers  (2024-03-11T10:50:53Z)
• Uncovering Bias in Face Generation Models [0.0]
  Recent advancements in GANs and diffusion models have enabled the creation of high-resolution, hyper-realistic images. These models may misrepresent certain social groups and present bias. This work is a novel analysis covering and embedding spaces for fine-grained understanding of bias over three approaches.
  arXiv  Detail & Related papers  (2023-02-22T18:57:35Z)
• Delving into Identify-Emphasize Paradigm for Combating Unknown Bias [52.76758938921129]
  We propose an effective bias-conflicting scoring method (ECS) to boost the identification accuracy. We also propose gradient alignment (GA) to balance the contributions of the mined bias-aligned and bias-conflicting samples. Experiments are conducted on multiple datasets in various settings, demonstrating that the proposed solution can mitigate the impact of unknown biases.
  arXiv  Detail & Related papers  (2023-02-22T14:50:24Z)
• Semi-FairVAE: Semi-supervised Fair Representation Learning with Adversarial Variational Autoencoder [92.67156911466397]
  We propose a semi-supervised fair representation learning approach based on adversarial variational autoencoder. We use a bias-aware model to capture inherent bias information on sensitive attribute. We also use a bias-free model to learn debiased fair representations by using adversarial learning to remove bias information from them.
  arXiv  Detail & Related papers  (2022-04-01T15:57:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.