AITTI: Learning Adaptive Inclusive Token for Text-to-Image Generation

• URL: http://arxiv.org/abs/2406.12805v2
• Date: Wed, 19 Jun 2024 02:28:33 GMT
• Title: AITTI: Learning Adaptive Inclusive Token for Text-to-Image Generation
• Authors: Xinyu Hou, Xiaoming Li, Chen Change Loy,
• Abstract summary: We learn adaptive inclusive tokens to shift the attribute distribution of the final generative outputs. Our method requires neither explicit attribute specification nor prior knowledge of the bias distribution. Our method achieves comparable performance to models that require specific attributes or editing directions for generation.
• Score: 53.65701943405546
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the high-quality results of text-to-image generation, stereotypical biases have been spotted in their generated contents, compromising the fairness of generative models. In this work, we propose to learn adaptive inclusive tokens to shift the attribute distribution of the final generative outputs. Unlike existing de-biasing approaches, our method requires neither explicit attribute specification nor prior knowledge of the bias distribution. Specifically, the core of our method is a lightweight adaptive mapping network, which can customize the inclusive tokens for the concepts to be de-biased, making the tokens generalizable to unseen concepts regardless of their original bias distributions. This is achieved by tuning the adaptive mapping network with a handful of balanced and inclusive samples using an anchor loss. Experimental results demonstrate that our method outperforms previous bias mitigation methods without attribute specification while preserving the alignment between generative results and text descriptions. Moreover, our method achieves comparable performance to models that require specific attributes or editing directions for generation. Extensive experiments showcase the effectiveness of our adaptive inclusive tokens in mitigating stereotypical bias in text-to-image generation. The code will be available at https://github.com/itsmag11/AITTI.

Related papers

• Latent Directions: A Simple Pathway to Bias Mitigation in Generative AI [45.54709270833219]
  Mitigating biases in generative AI and, particularly in text-to-image models, is of high importance given their growing implications in society. Our work introduces a novel approach to achieve diverse and inclusive synthetic images by learning a direction in the latent space. It is possible to linearly combine these learned latent directions to introduce new mitigations, and if desired, integrate it with text embedding adjustments.
  arXiv  Detail & Related papers  (2024-06-10T15:13:51Z)
• Mitigating Bias Using Model-Agnostic Data Attribution [2.9868610316099335]
  Mitigating bias in machine learning models is a critical endeavor for ensuring fairness and equity. We propose a novel approach to address bias by leveraging pixel image attributions to identify and regularize regions of images containing bias attributes.
  arXiv  Detail & Related papers  (2024-05-08T13:00:56Z)
• Distributionally Generative Augmentation for Fair Facial Attribute Classification [69.97710556164698]
  Facial Attribute Classification (FAC) holds substantial promise in widespread applications. FAC models trained by traditional methodologies can be unfair by exhibiting accuracy inconsistencies across varied data subpopulations. This work proposes a novel, generation-based two-stage framework to train a fair FAC model on biased data without additional annotation.
  arXiv  Detail & Related papers  (2024-03-11T10:50:53Z)
• Improving Fairness using Vision-Language Driven Image Augmentation [60.428157003498995]
  Fairness is crucial when training a deep-learning discriminative model, especially in the facial domain. Models tend to correlate specific characteristics (such as age and skin color) with unrelated attributes (downstream tasks) This paper proposes a method to mitigate these correlations to improve fairness.
  arXiv  Detail & Related papers  (2023-11-02T19:51:10Z)
• Debiasing Vision-Language Models via Biased Prompts [79.04467131711775]
  We propose a general approach for debiasing vision-language foundation models by projecting out biased directions in the text embedding. We show that debiasing only the text embedding with a calibrated projection matrix suffices to yield robust classifiers and fair generative models.
  arXiv  Detail & Related papers  (2023-01-31T20:09:33Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks [10.748627178113418]
  We propose a novel modular bias mitigation approach, consisting of stand-alone highly sparse debiasingworks. We conduct experiments on three classification tasks with gender, race, and age as protected attributes.
  arXiv  Detail & Related papers  (2022-05-30T15:21:25Z)
• Self-Conditioned Generative Adversarial Networks for Image Editing [61.50205580051405]
  Generative Adversarial Networks (GANs) are susceptible to bias, learned from either the unbalanced data, or through mode collapse. We argue that this bias is responsible not only for fairness concerns, but that it plays a key role in the collapse of latent-traversal editing methods when deviating away from the distribution's core.
  arXiv  Detail & Related papers  (2022-02-08T18:08:24Z)
• Matched sample selection with GANs for mitigating attribute confounding [30.488267816304177]
  We propose a matching approach that selects a subset of images from the full dataset with balanced attribute distributions across protected attributes. Our matching approach first projects real images onto a generative network's latent space in a manner that preserves semantic attributes. It then finds adversarial matches in this latent space across a chosen protected attribute, yielding a dataset where semantic and perceptual attributes are balanced across the protected attribute.
  arXiv  Detail & Related papers  (2021-03-24T19:18:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.