T2I-FineEval: Fine-Grained Compositional Metric for Text-to-Image Evaluation

• URL: http://arxiv.org/abs/2503.11481v1
• Date: Fri, 14 Mar 2025 15:06:12 GMT
• Title: T2I-FineEval: Fine-Grained Compositional Metric for Text-to-Image Evaluation
• Authors: Seyed Mohammad Hadi Hosseini, Amir Mohammad Izadi, Ali Abdollahi, Armin Saghafian, Mahdieh Soleymani Baghshah,
• Abstract summary: We propose a novel metric that breaks down images into components, and texts into fine-grained questions about the generated image for evaluation.<n>Our method outperforms previous state-of-the-art metrics, demonstrating its effectiveness in evaluating text-to-image generative models.
• Score: 2.273629240935727
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Although recent text-to-image generative models have achieved impressive performance, they still often struggle with capturing the compositional complexities of prompts including attribute binding, and spatial relationships between different entities. This misalignment is not revealed by common evaluation metrics such as CLIPScore. Recent works have proposed evaluation metrics that utilize Visual Question Answering (VQA) by decomposing prompts into questions about the generated image for more robust compositional evaluation. Although these methods align better with human evaluations, they still fail to fully cover the compositionality within the image. To address this, we propose a novel metric that breaks down images into components, and texts into fine-grained questions about the generated image for evaluation. Our method outperforms previous state-of-the-art metrics, demonstrating its effectiveness in evaluating text-to-image generative models. Code is available at https://github.com/hadi-hosseini/ T2I-FineEval.

Related papers

• Visual question answering based evaluation metrics for text-to-image generation [7.105786967332924]
  This paper proposes new evaluation metrics that assess the alignment between input text and generated images for every individual object. Experimental results show that our proposed evaluation approach is the superior metric that can simultaneously assess finer text-image alignment and image quality.
  arXiv  Detail & Related papers  (2024-11-15T13:32:23Z)
• Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models [16.00576040281808]
  We propose a novel framework called Image2Text2Image to evaluate image captioning models. A high similarity score suggests that the model has produced a faithful textual description, while a low score highlights discrepancies. Our framework does not rely on human-annotated captions reference, making it a valuable tool for assessing image captioning models.
  arXiv  Detail & Related papers  (2024-11-08T17:07:01Z)
• TypeScore: A Text Fidelity Metric for Text-to-Image Generative Models [39.06617653124486]
  We introduce a new evaluation framework called TypeScore to assess a model's ability to generate images with high-fidelity embedded text. Our proposed metric demonstrates greater resolution than CLIPScore to differentiate popular image generation models.
  arXiv  Detail & Related papers  (2024-11-02T07:56:54Z)
• A Novel Evaluation Framework for Image2Text Generation [15.10524860121122]
  We propose an evaluation framework rooted in a modern large language model (LLM) capable of image generation. A high similarity score suggests that the image captioning model has accurately generated textual descriptions. A low similarity score indicates discrepancies, revealing potential shortcomings in the model's performance.
  arXiv  Detail & Related papers  (2024-08-03T09:27:57Z)
• BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues [47.213906345208315]
  We propose BRIDGE, a new learnable and reference-free image captioning metric. Our proposal achieves state-of-the-art results compared to existing reference-free evaluation scores.
  arXiv  Detail & Related papers  (2024-07-29T18:00:17Z)
• GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation [103.3465421081531]
  VQAScore is a metric measuring the likelihood that a VQA model views an image as accurately depicting the prompt. Ranking by VQAScore is 2x to 3x more effective than other scoring methods like PickScore, HPSv2, and ImageReward. We release a new GenAI-Rank benchmark with over 40,000 human ratings to evaluate scoring metrics on ranking images generated from the same prompt.
  arXiv  Detail & Related papers  (2024-06-19T18:00:07Z)
• Likelihood-Based Text-to-Image Evaluation with Patch-Level Perceptual and Semantic Credit Assignment [48.835298314274254]
  We propose to evaluate text-to-image generation performance by directly estimating the likelihood of the generated images. A higher likelihood indicates better perceptual quality and better text-image alignment. It can successfully assess the generation ability of these models with as few as a hundred samples.
  arXiv  Detail & Related papers  (2023-08-16T17:26:47Z)
• T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation [55.16845189272573]
  T2I-CompBench++ is an enhanced benchmark for compositional text-to-image generation.<n>It comprises 8,000 compositional text prompts categorized into four primary groups: attribute binding, object relationships, generative numeracy, and complex compositions.
  arXiv  Detail & Related papers  (2023-07-12T17:59:42Z)
• TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering [86.38098280689027]
  We introduce an automatic evaluation metric that measures the faithfulness of a generated image to its text input via visual question answering (VQA) We present a comprehensive evaluation of existing text-to-image models using a benchmark consisting of 4K diverse text inputs and 25K questions across 12 categories (object, counting, etc.)
  arXiv  Detail & Related papers  (2023-03-21T14:41:02Z)
• Intrinsic Image Captioning Evaluation [53.51379676690971]
  We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE) Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
  arXiv  Detail & Related papers  (2020-12-14T08:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.