Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation

• URL: http://arxiv.org/abs/2608.29210v1
• Date: Sat, 29 Aug 2026 11:50:36 GMT
• Title: Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation
• Abstract summary: We introduce Imag-Eval, a benchmark to assess how Text-to-Image models ground compositional natural-language instructions into visual outputs.<n>Our benchmark comprises 1,140 prompts and 8,842 combined rules, and we evaluate it on several state-of-the-art models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-Image (T2I) models have recently achieved impressive visual fidelity, yet their evaluation remains constrained by benchmarks that are often difficult to interpret and insufficiently diagnostic. Existing skill-based evaluations tend to overlook critical failure modes that strongly impact usability but fall outside standard taxonomies, such as global incoherence arising from missing parts or physically implausible configurations (e.g., floating objects). In addition, prompt difficulty is typically controlled along a single dimension; either prompt length or the number of elements to generate. To address these limitations, we introduce Imag-Eval, a controlled benchmark designed to assess how T2I models ground compositional natural-language instructions into visual outputs. Unlike prior work that conflates surface linguistic complexity with compositional difficulty, Imag-Eval explicitly seeks to disentangles these factors by independently varying both the number of instances and the combination of constraints (rules), while avoiding error propagation. This design enables fine-grained and interpretable analysis of where cross-modal instruction following fails. Our benchmark comprises 1,140 prompts and 8,842 combined rules, and we evaluate it on several state-of-the-art models. Complementing this analysis with an additional study of over 2,000 prompts from a concurrent benchmark, our results suggest that, for structured skills, compositional difficulty is primarily governed by the number of grounded rules and their binding to instances,, rather than by prompt length alone.

Related papers

• DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models [78.62380562116135]
  We propose DynT2I-Eval, a fully automated dynamic evaluation framework for text-to-image (T2I) models.<n>It constructs a structured visual semantic space from long-form descriptions, decomposing prompts into controllable dimensions.<n>DynT2I-Eval evaluates model performance across text alignment, perceptual quality, and aesthetics.
  arXiv  Detail & Related papers  (2026-05-07T12:53:51Z)
• LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models [15.955490895662384]
  We introduce LIBERO-Para, a benchmark for fine-grained analysis of linguistic generalization.<n>We observe consistent performance degradation of 22-52 pp under paraphrasing.<n>We propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors.
  arXiv  Detail & Related papers  (2026-03-30T11:27:34Z)
• BaziQA-Benchmark: Evaluating Symbolic and Temporally Compositional Reasoning in Large Language Models [6.36932184701021]
  BaziQA-Benchmark is a standardized benchmark for evaluating symbolic and temporally compositional reasoning in large language models.<n>We evaluate contemporary language models under a multi-turn setting and analyze performance variation across temporal difficulty, reasoning domains, and inference protocols.<n>Results show that models consistently outperform chance but remain far from saturation, exhibiting pronounced sensitivity to temporal composition and reasoning order.
  arXiv  Detail & Related papers  (2026-02-13T12:45:42Z)
• SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis [16.02119252662268]
  We introduce SANEval (Spatial, Attribute, and Numeracy Evaluation), a comprehensive benchmark for open-vocabulary compositional evaluation.<n> SANEval combines a large language model (LLM) for deep prompt understanding with an LLM-enhanced, open-vocabulary object detector to robustly evaluate compositional adherence.<n>Our metric achieves a Spearman's rank correlation with statistically different results than those of existing benchmarks across tasks of attribute binding, spatial relations, and numeracy.
  arXiv  Detail & Related papers  (2026-01-30T19:10:20Z)
• Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play? [63.66192651248858]
  T2I-CoReBench is a comprehensive and complex benchmark that evaluates both composition and reasoning capabilities of T2I models.<n>To increase complexity, driven by the inherent real-world complexities, we curate each prompt with higher compositional density for composition.<n>In statistics, our benchmark comprises 1,080 challenging prompts and around 13,500 checklist questions.
  arXiv  Detail & Related papers  (2025-09-03T17:58:12Z)
• A Controllable Examination for Long-Context Language Models [62.845852724511964]
  This study introduces $textbfLongBioBench, a benchmark for evaluating long-context language models.<n>We show that most models still exhibit deficiencies in semantic understanding and elementary reasoning over retrieved results.<n>Our further analysis indicates some design choices employed by existing synthetic benchmarks, such as contextual non-coherence.
  arXiv  Detail & Related papers  (2025-06-03T14:23:06Z)
• A linguistically-motivated evaluation methodology for unraveling model's abilities in reading comprehension tasks [10.181408678232055]
  We introduce an evaluation methodology for reading comprehension tasks based on the intuition that certain examples consistently yield lower scores regardless of model size or architecture.<n>We capitalize on semantic frame annotation for characterizing this complexity, and study seven complexity factors that may account for model's difficulty.
  arXiv  Detail & Related papers  (2025-01-29T11:05:20Z)
• StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs [78.84060166851805]
  StructTest is a novel benchmark that evaluates large language models (LLMs) on their ability to follow compositional instructions and generate structured outputs.<n> Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets.<n>We demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o.
  arXiv  Detail & Related papers  (2024-12-23T22:08:40Z)
• SHINE: Saliency-aware HIerarchical NEgative Ranking for Compositional Temporal Grounding [52.98133831401225]
  Temporal grounding, also known as video moment retrieval, aims at locating video segments corresponding to a given query sentence. We propose a large language model-driven method for negative query construction, utilizing GPT-3.5-Turbo. We introduce a coarse-to-fine saliency ranking strategy, which encourages the model to learn the multi-granularity semantic relationships between videos and hierarchical negative queries.
  arXiv  Detail & Related papers  (2024-07-06T16:08:17Z)
• Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2) [62.44395685571094]
  We introduce T2IScoreScore, a curated set of semantic error graphs containing a prompt and a set of increasingly erroneous images. These allow us to rigorously judge whether a given prompt faithfulness metric can correctly order images with respect to their objective error count. We find that the state-of-the-art VLM-based metrics fail to significantly outperform simple (and supposedly worse) feature-based metrics like CLIPScore.
  arXiv  Detail & Related papers  (2024-04-05T17:57:16Z)
• SOUL: Towards Sentiment and Opinion Understanding of Language [96.74878032417054]
  We propose a new task called Sentiment and Opinion Understanding of Language (SOUL) SOUL aims to evaluate sentiment understanding through two subtasks: Review (RC) and Justification Generation (JG)
  arXiv  Detail & Related papers  (2023-10-27T06:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.