Analyzing Quality, Bias, and Performance in Text-to-Image Generative Models

• URL: http://arxiv.org/abs/2407.00138v1
• Date: Fri, 28 Jun 2024 14:10:42 GMT
• Title: Analyzing Quality, Bias, and Performance in Text-to-Image Generative Models
• Authors: Nila Masrourisaadat, Nazanin Sedaghatkish, Fatemeh Sarshartehrani, Edward A. Fox,
• Abstract summary: Despite advances in generative models, most studies ignore the presence of bias. In this paper, we examine several text-to-image models not only by qualitatively assessing their performance in generating accurate images of human faces, groups, and specified numbers of objects but also by presenting a social bias analysis. As expected, models with larger capacity generate higher-quality images. However, we also document the inherent gender or social biases these models possess, offering a more complete understanding of their impact and limitations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advances in generative models have led to significant interest in image synthesis, demonstrating the ability to generate high-quality images for a diverse range of text prompts. Despite this progress, most studies ignore the presence of bias. In this paper, we examine several text-to-image models not only by qualitatively assessing their performance in generating accurate images of human faces, groups, and specified numbers of objects but also by presenting a social bias analysis. As expected, models with larger capacity generate higher-quality images. However, we also document the inherent gender or social biases these models possess, offering a more complete understanding of their impact and limitations.

Related papers

• KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities [93.74881034001312]
  We conduct a systematic study on the fidelity of entities in text-to-image generation models. We focus on their ability to generate a wide range of real-world visual entities, such as landmark buildings, aircraft, plants, and animals. Our findings reveal that even the most advanced text-to-image models often fail to generate entities with accurate visual details.
  arXiv  Detail & Related papers  (2024-10-15T17:50:37Z)
• Gender Bias Evaluation in Text-to-image Generation: A Survey [25.702257177921048]
  We review recent work on gender bias evaluation in text-to-image generation. We focus on the evaluation of recent popular models such as Stable Diffusion and DALL-E 2.
  arXiv  Detail & Related papers  (2024-08-21T06:01:23Z)
• Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation [87.50120181861362]
  VisionPrefer is a high-quality and fine-grained preference dataset that captures multiple preference aspects. We train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators.
  arXiv  Detail & Related papers  (2024-04-23T14:53:15Z)
• TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models [22.076898042211305]
  We propose a general approach to study and quantify a broad spectrum of biases, for any TTI model and for any prompt. Our approach automatically identifies potential biases that might be relevant to the given prompt, and measures those biases. We show that our method is uniquely capable of explaining complex multi-dimensional biases through semantic concepts.
  arXiv  Detail & Related papers  (2023-12-03T02:31:37Z)
• Situating the social issues of image generation models in the model life cycle: a sociotechnical approach [20.99805435959377]
  This paper reports on a novel, comprehensive categorization of the social issues associated with image generation models. We identify seven issue clusters arising from image generation models: data issues, intellectual property, bias, privacy, and the impacts on the informational, cultural, and natural environments. We argue that the risks posed by image generation models are comparable in severity to the risks posed by large language models.
  arXiv  Detail & Related papers  (2023-11-30T08:32:32Z)
• Holistic Evaluation of Text-To-Image Models [153.47415461488097]
  We introduce a new benchmark, Holistic Evaluation of Text-to-Image Models (HEIM) We identify 12 aspects, including text-image alignment, image quality, aesthetics, originality, reasoning, knowledge, bias, toxicity, fairness, robustness, multilinguality, and efficiency. Our results reveal that no single model excels in all aspects, with different models demonstrating different strengths.
  arXiv  Detail & Related papers  (2023-11-07T19:00:56Z)
• Limitations of Face Image Generation [12.11955119100926]
  We study the efficacy and shortcomings of generative models in the context of face generation. We identify several limitations of face image generation that include faithfulness to the text prompt, demographic disparities, and distributional shifts. We present an analytical model that provides insights into how training data selection contributes to the performance of generative models.
  arXiv  Detail & Related papers  (2023-09-13T19:33:26Z)
• Auditing Gender Presentation Differences in Text-to-Image Models [54.16959473093973]
  We study how gender is presented differently in text-to-image models. By probing gender indicators in the input text, we quantify the frequency differences of presentation-centric attributes. We propose an automatic method to estimate such differences.
  arXiv  Detail & Related papers  (2023-02-07T18:52:22Z)
• DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models [73.12069620086311]
  We investigate the visual reasoning capabilities and social biases of text-to-image models. First, we measure three visual reasoning skills: object recognition, object counting, and spatial relation understanding. Second, we assess the gender and skin tone biases by measuring the gender/skin tone distribution of generated images.
  arXiv  Detail & Related papers  (2022-02-08T18:36:52Z)
• Unravelling the Effect of Image Distortions for Biased Prediction of Pre-trained Face Recognition Models [86.79402670904338]
  We evaluate the performance of four state-of-the-art deep face recognition models in the presence of image distortions. We have observed that image distortions have a relationship with the performance gap of the model across different subgroups.
  arXiv  Detail & Related papers  (2021-08-14T16:49:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.