ProImage-Bench: Rubric-Based Evaluation for Professional Image Generation

• URL: http://arxiv.org/abs/2512.12220v1
• Date: Sat, 13 Dec 2025 07:13:43 GMT
• Title: ProImage-Bench: Rubric-Based Evaluation for Professional Image Generation
• Abstract summary: We study professional image generation, where a model must synthesize scientifically precise illustrations from technical descriptions.<n>For 654 figures collected from real textbooks and technical reports, we construct detailed image instructions and a hierarchy of rubrics that decompose correctness into 6,076 criteria and 44,131 binary checks.<n>We benchmark several representative text-to-image models on ProImage-Bench and find that, despite strong open-domain performance, the best base model reaches only 0.791 accuracy and 0.553 criterion score overall.
• Score: 151.75112778479468
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We study professional image generation, where a model must synthesize information-dense, scientifically precise illustrations from technical descriptions rather than merely produce visually plausible pictures. To quantify the progress, we introduce ProImage-Bench, a rubric-based benchmark that targets biology schematics, engineering/patent drawings, and general scientific diagrams. For 654 figures collected from real textbooks and technical reports, we construct detailed image instructions and a hierarchy of rubrics that decompose correctness into 6,076 criteria and 44,131 binary checks. Rubrics are derived from surrounding text and reference figures using large multimodal models, and are evaluated by an automated LMM-based judge with a principled penalty scheme that aggregates sub-question outcomes into interpretable criterion scores. We benchmark several representative text-to-image models on ProImage-Bench and find that, despite strong open-domain performance, the best base model reaches only 0.791 rubric accuracy and 0.553 criterion score overall, revealing substantial gaps in fine-grained scientific fidelity. Finally, we show that the same rubrics provide actionable supervision: feeding failed checks back into an editing model for iterative refinement boosts a strong generator from 0.653 to 0.865 in rubric accuracy and from 0.388 to 0.697 in criterion score. ProImage-Bench thus offers both a rigorous diagnostic for professional image generation and a scalable signal for improving specification-faithful scientific illustrations.

Related papers

• Are Image Generators Zero-Shot Perceivers? A Rigorous Evaluation [54.18771863630389]
  We ask how far image generators can go on public visual perception benchmarks in a zero-shot setting.<n>We introduce ProbeGen, a benchmark for zero-shot generative perception that casts monocular depth estimation.<n>We observe that pretrained image generators show measurable zero-shot perceptual competence, but with a clear trade-off.
  arXiv  Detail & Related papers  (2026-09-07T18:48:57Z)
• TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation [74.69068352546073]
  TRACE-Bench is an evaluation tool for multi-reference images spanning diverse artistic styles and real-world subjects.<n>It is built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects.<n>Its formula structure drives an operator-aligned evaluation protocol for per-capability scoring and a diagnostic tree analysis.
  arXiv  Detail & Related papers  (2026-08-17T16:15:50Z)
• SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context [27.236953887247125]
  We propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions.<n>The data covers top computer-science conferences from 2020 to 2025.<n>Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context.
  arXiv  Detail & Related papers  (2026-07-29T16:07:44Z)
• Qwen-Image-2.0-RL Technical Report [66.2346321737379]
  We present Qwen-Image-2.0-RL, a post-training pipeline that applies reinforcement learning from human feedback and on-policy distillation.<n>Qwen-Image-2.0-RL achieves 57.84 overall score on Qwen-Image-Bench, Elo ratings of 1193 in text-to-image arena and 1349 in image edit arena.
  arXiv  Detail & Related papers  (2026-06-25T23:49:50Z)
• Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation [52.930280794143876]
  We introduce Qwen-Image-Bench, a creator-centric benchmark co-designed with professional artists and grounded in real-world creation scenarios.<n>We organize these five pillars into a top-down hierarchical taxonomy that decomposes into 23 second-level sub-capabilities and 56 third-level rubrics.<n>We train a unified judge model Q-Judger based on Qwen3.6-27B, supervised by 80 professional annotators from global art academies under blind labeling and triple-review protocols.
  arXiv  Detail & Related papers  (2026-05-27T07:46:43Z)
• Medical thinking with multiple images [44.04557445622649]
  We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images.<n>The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case.<n>On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy.
  arXiv  Detail & Related papers  (2026-04-14T18:51:07Z)
• SCHEMA for Gemini 3 Pro Image: A Structured Methodology for Controlled AI Image Generation on Google's Native Multimodal Model [0.0]
  SCHEMA is an engineered framework built on systematic professional practice encompassing 850 verified API predictions.<n>SchEMA scales practitioner control from exploratory (approximately 5%) to directive (approximately 95%)<n>Key findings include an observed 91% Mandatory compliance rate and 94% Prohibitions compliance rate across 621 structured prompts.
  arXiv  Detail & Related papers  (2026-02-21T16:51:40Z)
• GenExam: A Multidisciplinary Text-to-Image Exam [91.06661449186537]
  GenExam is the first benchmark for multidisciplinary text-to-image exams.<n>It features 1,000 samples across 10 subjects with exam-style prompts organized under a four-level taxonomy.<n>Each problem is equipped with ground-truth images and fine-grained scoring points.
  arXiv  Detail & Related papers  (2025-09-17T17:59:14Z)
• FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark [62.2297310186354]
  FLUX-Reason-6M is a massive dataset consisting of 6 million high-quality FLUX-generated images and 20 million bilingual (English and Chinese) descriptions.<n> PRISM-Bench offers a novel evaluation standard with seven distinct tracks, including a formidable Long Text challenge using GCoT.
  arXiv  Detail & Related papers  (2025-09-11T17:59:59Z)
• Edge-Enhanced Vision Transformer Framework for Accurate AI-Generated Image Detection [0.0]
  We propose a hybrid detection framework that combines a fine-tuned Vision Transformer (ViT) with a novel edge-based image processing module.<n>The proposed method is highly suitable for real-world applications in automated content verification and digital forensics.
  arXiv  Detail & Related papers  (2025-08-25T10:30:56Z)
• A Unified Agentic Framework for Evaluating Conditional Image Generation [66.25099219134441]
  Conditional image generation has gained significant attention for its ability to personalize content.<n>This paper introduces CIGEval, a unified agentic framework for comprehensive evaluation of conditional image generation tasks.
  arXiv  Detail & Related papers  (2025-04-09T17:04:14Z)
• RELICT: A Replica Detection Framework for Medical Image Generation [34.82692226532414]
  memorization in generative models can lead to unintended leakage of sensitive patient information and limit model utility.<n>We propose a framework for identifying replicas, i.e. nearly identical copies of the training data, in synthetic medical image datasets.
  arXiv  Detail & Related papers  (2025-02-24T17:37:19Z)
• ImagiNet: A Multi-Content Benchmark for Synthetic Image Detection [0.0]
  We introduce ImagiNet, a dataset of 200K examples spanning four categories: photos, paintings, faces, and miscellaneous.<n>Synthetic images in ImagiNet are produced with both open-source and proprietary generators, whereas real counterparts for each content type are collected from public datasets.
  arXiv  Detail & Related papers  (2024-07-29T13:57:24Z)
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation [52.509092010267665]
  We introduce LlamaGen, a new family of image generation models that apply original next-token prediction'' paradigm of large language models to visual generation domain. It is an affirmative answer to whether vanilla autoregressive models, e.g., Llama, without inductive biases on visual signals can achieve state-of-the-art image generation performance if scaling properly.
  arXiv  Detail & Related papers  (2024-06-10T17:59:52Z)
• ImagenHub: Standardizing the evaluation of conditional image generation models [48.51117156168]
  This paper proposes ImagenHub, which is a one-stop library to standardize the inference and evaluation of all conditional image generation models. We design two human evaluation scores, i.e. Semantic Consistency and Perceptual Quality, along with comprehensive guidelines to evaluate generated images. Our human evaluation achieves a high inter-worker agreement of Krippendorff's alpha on 76% models with a value higher than 0.4.
  arXiv  Detail & Related papers  (2023-10-02T19:41:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.