Multi-Group Proportional Representation for Text-to-Image Models

• URL: http://arxiv.org/abs/2505.24023v1
• Date: Thu, 29 May 2025 21:48:28 GMT
• Title: Multi-Group Proportional Representation for Text-to-Image Models
• Authors: Sangwon Jung, Alex Oesterling, Claudio Mayrink Verdun, Sajani Vithana, Taesup Moon, Flavio P. Calmon,
• Abstract summary: Text-to-image (T2I) generative models can create vivid, realistic images from textual descriptions.<n>As these models proliferate, they expose new concerns about their ability to represent diverse demographic groups, propagate stereotypes, and efface minority populations.<n>This paper introduces a novel framework to measure the representation of intersectional groups in images generated by T2I models by applying the Multi-Group Proportional Representation metric.
• Score: 19.36512604668349
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image (T2I) generative models can create vivid, realistic images from textual descriptions. As these models proliferate, they expose new concerns about their ability to represent diverse demographic groups, propagate stereotypes, and efface minority populations. Despite growing attention to the "safe" and "responsible" design of artificial intelligence (AI), there is no established methodology to systematically measure and control representational harms in image generation. This paper introduces a novel framework to measure the representation of intersectional groups in images generated by T2I models by applying the Multi-Group Proportional Representation (MPR) metric. MPR evaluates the worst-case deviation of representation statistics across given population groups in images produced by a generative model, allowing for flexible and context-specific measurements based on user requirements. We also develop an algorithm to optimize T2I models for this metric. Through experiments, we demonstrate that MPR can effectively measure representation statistics across multiple intersectional groups and, when used as a training objective, can guide models toward a more balanced generation across demographic groups while maintaining generation quality.

Related papers

• UniVG: A Generalist Diffusion Model for Unified Image Generation and Editing [59.590505989071175]
  Text-to-Image (T2I) diffusion models have shown impressive results in generating visually compelling images following user prompts.<n>We introduce UniVG, a generalist diffusion model capable of supporting a diverse range of image generation tasks with a single set of weights.
  arXiv  Detail & Related papers  (2025-03-16T21:11:25Z)
• Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models [54.052963634384945]
  We introduce the Image Regeneration task to assess text-to-image models. We use GPT4V to bridge the gap between the reference image and the text input for the T2I model. We also present ImageRepainter framework to enhance the quality of generated images.
  arXiv  Detail & Related papers  (2024-11-14T13:52:43Z)
• VLEU: a Method for Automatic Evaluation for Generalizability of Text-to-Image Models [18.259733507395634]
  We introduce a new metric called Visual Language Evaluation Understudy (VLEU) VLEU quantifies a model's generalizability by computing the Kullback-Leibler divergence between the marginal distribution of the visual text and the conditional distribution of the images generated by the model. Our experiments demonstrate the effectiveness of VLEU in evaluating the generalization capability of various T2I models.
  arXiv  Detail & Related papers  (2024-09-23T04:50:36Z)
• Reinforcing Pre-trained Models Using Counterfactual Images [54.26310919385808]
  This paper proposes a novel framework to reinforce classification models using language-guided generated counterfactual images. We identify model weaknesses by testing the model using the counterfactual image dataset. We employ the counterfactual images as an augmented dataset to fine-tune and reinforce the classification model.
  arXiv  Detail & Related papers  (2024-06-19T08:07:14Z)
• Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation [87.50120181861362]
  VisionPrefer is a high-quality and fine-grained preference dataset that captures multiple preference aspects. We train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators.
  arXiv  Detail & Related papers  (2024-04-23T14:53:15Z)
• FairRAG: Fair Human Generation via Fair Retrieval Augmentation [27.069276012884398]
  We introduce Fair Retrieval Augmented Generation (FairRAG), a novel framework that conditions pre-trained generative models on reference images retrieved from an external image database to improve fairness in human generation. To enhance fairness, FairRAG applies simple-yet-effective debiasing strategies, providing images from diverse demographic groups during the generative process.
  arXiv  Detail & Related papers  (2024-03-29T03:56:19Z)
• T-HITL Effectively Addresses Problematic Associations in Image Generation and Maintains Overall Visual Quality [52.5529784801908]
  We focus on addressing the generation of problematic associations between demographic groups and semantic concepts. We propose a new methodology with twice-human-in-the-loop (T-HITL) that promises improvements in both reducing problematic associations and also maintaining visual quality.
  arXiv  Detail & Related papers  (2024-02-27T00:29:33Z)
• DeAR: Debiasing Vision-Language Models with Additive Residuals [5.672132510411465]
  Large pre-trained vision-language models (VLMs) provide rich, adaptable image and text representations. These models suffer from societal biases owing to the skewed distribution of various identity groups in the training data. We present DeAR, a novel debiasing method that learns additive residual image representations to offset the original representations.
  arXiv  Detail & Related papers  (2023-03-18T14:57:43Z)
• MAUVE Scores for Generative Models: Theory and Practice [95.86006777961182]
  We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images. We find that MAUVE can quantify the gaps between the distributions of human-written text and those of modern neural language models. We demonstrate in the vision domain that MAUVE can identify known properties of generated images on par with or better than existing metrics.
  arXiv  Detail & Related papers  (2022-12-30T07:37:40Z)
• Mutual Information Divergence: A Unified Metric for Multimodal Generative Models [19.520177195241704]
  We propose the negative Gaussian cross-mutual information using the CLIP features as a unified metric, coined by Mutual Information Divergence (MID) We extensively compare it with competing metrics using carefully-generated or human-annotated judgments in text-to-image generation and image captioning tasks. The proposed MID significantly outperforms the competitive methods by having consistency across benchmarks, sample parsimony, and robustness toward the exploited CLIP model.
  arXiv  Detail & Related papers  (2022-05-25T09:34:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.