VISTAR:A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation

• URL: http://arxiv.org/abs/2508.06152v1
• Date: Fri, 08 Aug 2025 09:15:02 GMT
• Title: VISTAR:A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation
• Abstract summary: We present VISTAR, a user-centric, multi-dimensional benchmark for text-to-image (T2I) evaluation.<n>It employs deterministic, scriptable metrics for physically quantifiable attributes.<n>The benchmark comprises 2,845 prompts validated by over 15,000 human pairwise comparisons.
• Score: 15.513836968440053
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present VISTAR, a user-centric, multi-dimensional benchmark for text-to-image (T2I) evaluation that addresses the limitations of existing metrics. VISTAR introduces a two-tier hybrid paradigm: it employs deterministic, scriptable metrics for physically quantifiable attributes (e.g., text rendering, lighting) and a novel Hierarchical Weighted P/N Questioning (HWPQ) scheme that uses constrained vision-language models to assess abstract semantics (e.g., style fusion, cultural fidelity). Grounded in a Delphi study with 120 experts, we defined seven user roles and nine evaluation angles to construct the benchmark, which comprises 2,845 prompts validated by over 15,000 human pairwise comparisons. Our metrics achieve high human alignment (>75%), with the HWPQ scheme reaching 85.9% accuracy on abstract semantics, significantly outperforming VQA baselines. Comprehensive evaluation of state-of-the-art models reveals no universal champion, as role-weighted scores reorder rankings and provide actionable guidance for domain-specific deployment. All resources are publicly released to foster reproducible T2I assessment.

Related papers

• UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation [40.644151228285246]
  We introduce UniGenBench++, a unified semantic assessment benchmark for text-to-image generation.<n>It comprises 600 prompts organized hierarchically to ensure both coverage and efficiency.<n>It provides both English and Chinese versions of each prompt in short and long forms.
  arXiv  Detail & Related papers  (2025-10-21T14:56:46Z)
• TIT-Score: Evaluating Long-Prompt Based Text-to-Image Alignment via Text-to-Image-to-Text Consistency [81.17906057429329]
  LPG-Bench is a comprehensive benchmark for evaluating long-prompt-based text-to-image generation.<n>We generate 2,600 images from 13 state-of-the-art models and perform comprehensive human-ranked annotations.<n>We introduce a novel zero-shot metric based on text-to-image-to-text consistency, termed TIT, for evaluating long-prompt-generated images.
  arXiv  Detail & Related papers  (2025-10-03T13:25:16Z)
• MEF: A Systematic Evaluation Framework for Text-to-Image Models [21.006921005280493]
  Current evaluations rely on either ELO for overall ranking or MOS for dimension-specific scoring.<n>We introduce the Magic Evaluation Framework (MEF), a systematic and practical approach for evaluating T2I models.<n>We release our evaluation framework and make Magic-Bench-377 fully open-source to advance research in the evaluation of visual generative models.
  arXiv  Detail & Related papers  (2025-09-22T15:32:42Z)
• Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content [53.680220823229895]
  evaluating text-to-vision content hinges on two crucial aspects: visual quality and alignment.<n>We introduce a dataset designed to Evaluate Visual quality and Alignment Level for text-to-vision content (Q-EVAL-100K)<n>We propose Q-Eval-Score, a unified model capable of evaluating both visual quality and alignment with special improvements for handling long-text prompt alignment.
  arXiv  Detail & Related papers  (2025-03-04T07:28:45Z)
• EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation [29.176750442205325]
  In this study, we contribute an EvalMuse-40K benchmark, gathering 40K image-text pairs with fine-grained human annotations for image-text alignment-related tasks.<n>We introduce two new methods to evaluate the image-text alignment capabilities of T2I models.
  arXiv  Detail & Related papers  (2024-12-24T04:08:25Z)
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models [94.31327813151208]
  BiGGen Bench is a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks.<n>A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation.
  arXiv  Detail & Related papers  (2024-06-09T12:30:30Z)
• Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings [32.77190832269137]
  We introduce a skills-based benchmark that can discriminate models across different human templates.<n>We gather human ratings across four templates and four T2I models for a total of >100K annotations.<n>We introduce a new QA-based auto-eval metric that is better correlated with human ratings than existing metrics.
  arXiv  Detail & Related papers  (2024-04-25T17:58:43Z)
• A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metrics [58.83242220266935]
  We introduce Winoground-T2I, a benchmark designed to evaluate the compositionality of T2I models. This benchmark includes 11K complex, high-quality contrastive sentence pairs spanning 20 categories. We use Winoground-T2I with a dual objective: to evaluate the performance of T2I models and the metrics used for their evaluation.
  arXiv  Detail & Related papers  (2023-12-04T20:47:48Z)
• VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use [49.574651930395305]
  VisIT-Bench is a benchmark for evaluation of instruction-following vision-language models. Our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. We quantify quality gaps between models and references using both human and automatic evaluations.
  arXiv  Detail & Related papers  (2023-08-12T15:27:51Z)
• INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback [80.57617091714448]
  We present InstructScore, an explainable evaluation metric for text generation. We fine-tune a text evaluation metric based on LLaMA, producing a score for generated text and a human readable diagnostic report.
  arXiv  Detail & Related papers  (2023-05-23T17:27:22Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.