Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models

• URL: http://arxiv.org/abs/2606.05949v2
• Date: Fri, 05 Jun 2026 06:49:31 GMT
• Title: Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models
• Abstract summary: We introduce FEPBench, a benchmark built from carefully selected high-quality scientific illustrations.<n>We evaluate T2I models along three dimensions: instruction faithfulness, reasoning enrichment, and semantic precision.<n>Results show that even state-of-the-art closed-source models, such as GPT Image 2 and Nano Banana Pro, still suffer from text-rendering bottlenecks, limited reasoning enrichment, and difficulty balancing generation richness with precision.
• Score: 49.93401412148884
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Scientific illustrations are essential tools for communicating research findings, especially in natural science, where they visualize complex concepts and processes. As Text-to-Image (T2I) models become increasingly capable, researchers have started to use them for scientific illustration generation. However, existing benchmarks often assess outputs at a holistic level, overlooking fine-grained elements, while scientific reasoning ability and output conciseness remain under-quantified. We introduce FEPBench, a benchmark built from carefully selected high-quality scientific illustrations across multiple disciplines and layout types. With the assistance of multimodal large language models (MLLMs) and human experts, we provide fine-grained atom set annotations and systematically evaluate T2I models along three dimensions: instruction faithfulness, reasoning enrichment, and semantic precision. Our evaluation further decomposes model performance across visual, textual, relation, and layout elements. Results show that even state-of-the-art (SOTA) closed-source models, such as GPT Image 2 and Nano Banana Pro, still suffer from text-rendering bottlenecks, limited reasoning enrichment, and difficulty balancing generation richness with precision. These findings provide practical guidance for improving and deploying T2I models in scientific illustration generation. Benchmark data, atom set annotations, and evaluation code will be released by us.

Related papers

• From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding [0.0]
  Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images.<n>We introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula.<n>We construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences.
  arXiv  Detail & Related papers  (2026-09-01T09:07:31Z)
• Benchmarking and Enhancing Text-to-Image Models for Generating Visual Representations in Early Arithmetic Education [57.241885377686486]
  We introduce equation-to-visual generation, a task that requires producing meaningful visuals from arithmetic equations.<n>We construct E2V-Bench, a benchmark spanning four pedagogically grounded visual types, along with automatic metrics for evaluating visual correctness.<n>Our evaluation reveals that recent text-to-image (T2I) models frequently fail on this task, with errors dominated by incorrect object counts and broken relational structure.
  arXiv  Detail & Related papers  (2026-05-29T12:18:08Z)
• Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learning [36.86798995699176]
  We present a closed-loop framework featuring SciTikZ-230K, a large-scale, high-quality dataset from our Execution-Centric Data Engine.<n>Our trained model SciTikZer-8B achieves state-of-the-art performance, consistently outperforming proprietary giants like Gemini-2.5-Pro and massive models like Qwen3-VL-235B-A22B-Instruct.
  arXiv  Detail & Related papers  (2026-04-07T16:58:14Z)
• Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility [57.83550091882176]
  We study scientific image synthesis across generation paradigms, evaluation, and downstream use.<n>We introduce SciGenBench, which evaluates generated images based on information utility and logical validity.<n>We show that fine-tuning Large Multimodal Models on rigorously verified synthetic scientific images yields consistent reasoning gains.
  arXiv  Detail & Related papers  (2026-01-17T14:18:36Z)
• AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models [58.85362281293525]
  We introduce AcT2I, a benchmark designed to evaluate the performance of T2I models in generating images from action-centric prompts.<n>We experimentally validate that leading T2I models do not fare well on AcT2I.<n>We build upon this by developing a training-free, knowledge distillation technique utilizing Large Language Models to address this limitation.
  arXiv  Detail & Related papers  (2025-09-19T16:41:39Z)
• SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model [21.81341169834812]
  SridBench is the first benchmark for scientific figure generation.<n>It comprises 1,120 instances from leading scientific papers across 13 natural and computer science disciplines.<n>Results reveal that even top-tier models like GPT-4o-image lag behind human performance.
  arXiv  Detail & Related papers  (2025-05-28T08:51:01Z)
• PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models [50.33699462106502]
  Text-to-image (T2I) models frequently fail to produce images consistent with physical commonsense. Current T2I evaluation benchmarks focus on metrics such as accuracy, bias, and safety, neglecting the evaluation of models' internal knowledge. We introduce PhyBench, a comprehensive T2I evaluation dataset comprising 700 prompts across 4 primary categories: mechanics, optics, thermodynamics, and material properties.
  arXiv  Detail & Related papers  (2024-06-17T17:49:01Z)
• Benchmarking Spatial Relationships in Text-to-Image Generation [102.62422723894232]
  We investigate the ability of text-to-image models to generate correct spatial relationships among objects. We present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image. Our experiments reveal a surprising finding that, although state-of-the-art T2I models exhibit high image quality, they are severely limited in their ability to generate multiple objects or the specified spatial relations between them.
  arXiv  Detail & Related papers  (2022-12-20T06:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.