GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models

• URL: http://arxiv.org/abs/2604.04172v1
• Date: Sun, 05 Apr 2026 16:30:21 GMT
• Title: GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models
• Abstract summary: We introduce GENFIG1, a benchmark for generative AI models (e.g., Vision-Language Models)<n> GENFIG1 evaluates models for their ability to produce figures that clearly express and motivate the central idea of a paper.<n>We evaluate a suite of representative models on GENFIG1 and demonstrate that the task presents significant challenges, even for the best-performing systems.
• Score: 39.79177783850843
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many science papers, "Figure 1" serves as the primary visual summary of the core research idea. These figures are visually simple yet conceptually rich, often requiring significant effort and iteration by human authors to get right, highlighting the difficulty of science visual communication. With this intuition, we introduce GENFIG1, a benchmark for generative AI models (e.g., Vision-Language Models). GENFIG1 evaluates models for their ability to produce figures that clearly express and motivate the central idea of a paper (title, abstract, introduction, and figure caption) as input. Solving GENFIG1 requires more than producing visually appealing graphics: the task entails reasoning for text-to-image generation that couples scientific understanding with visual synthesis. Specifically, models must (i) comprehend and grasp the technical concepts of the paper, (ii) identify the most salient ones, and (iii) design a coherent and aesthetically effective graphic that conveys those concepts visually and is faithful to the input. We curate the benchmark from papers published at top deep-learning conferences, apply stringent quality control, and introduce an automatic evaluation metric that correlates well with expert human judgments. We evaluate a suite of representative models on GENFIG1 and demonstrate that the task presents significant challenges, even for the best-performing systems. We hope this benchmark serves as a foundation for future progress in multimodal AI.

Related papers

• From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding [0.0]
  Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images.<n>We introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula.<n>We construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences.
  arXiv  Detail & Related papers  (2026-09-01T09:07:31Z)
• Visual General Intelligence: A White Paper [150.33091833511565]
  This paper reconsiders intelligence from a vision-centered perspective.<n>It examines whether intelligence emerging from visual experience and learning may provide a pathway toward AGI.
  arXiv  Detail & Related papers  (2026-08-26T15:36:19Z)
• Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models [49.93401412148884]
  We introduce FEPBench, a benchmark built from carefully selected high-quality scientific illustrations.<n>We evaluate T2I models along three dimensions: instruction faithfulness, reasoning enrichment, and semantic precision.<n>Results show that even state-of-the-art closed-source models, such as GPT Image 2 and Nano Banana Pro, still suffer from text-rendering bottlenecks, limited reasoning enrichment, and difficulty balancing generation richness with precision.
  arXiv  Detail & Related papers  (2026-06-04T09:49:02Z)
• LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation [57.70907667652966]
  Introduction writing is especially challenging, which demands logical soundness and verifiable faithfulness.<n>Most AI-assisted methods treat the task as text generation instead of reasoning and structuring, leading to severe drawbacks.<n>We propose LECTOR, a novel Logic-Expression Co-Reinforcement Learning framework that can strictly follow the scientist's logic, add high-quality citations and keep structured expressions.
  arXiv  Detail & Related papers  (2026-05-25T15:41:16Z)
• Toward a Machine Bertin: Why Visualization Needs Design Principles for Machine Cognition [0.27074235008521247]
  Vision-language models (VLMs) increasingly consume chart images in automated analysis pipelines.<n>Current approaches address this gap primarily by bypassing vision entirely.<n>This paper makes the case that the visualization field needs to investigate machine-oriented visual design as a distinct research problem.
  arXiv  Detail & Related papers  (2026-02-02T01:39:33Z)
• GenExam: A Multidisciplinary Text-to-Image Exam [91.06661449186537]
  GenExam is the first benchmark for multidisciplinary text-to-image exams.<n>It features 1,000 samples across 10 subjects with exam-style prompts organized under a four-level taxonomy.<n>Each problem is equipped with ground-truth images and fine-grained scoring points.
  arXiv  Detail & Related papers  (2025-09-17T17:59:14Z)
• Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models [51.900488744931785]
  We introduce the Visual Graph Arena (VGA) to evaluate and improve AI systems' capacity for visual abstraction.<n>Humans achieve near-perfect accuracy across tasks, while models totally failed on isomorphism detection and showed limited success in path/cycle tasks.<n>By isolating the challenge of representation-invariant reasoning, the VGA provides a framework to drive progress toward human-like conceptualization in AI visual models.
  arXiv  Detail & Related papers  (2025-06-06T17:06:25Z)
• SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model [21.81341169834812]
  SridBench is the first benchmark for scientific figure generation.<n>It comprises 1,120 instances from leading scientific papers across 13 natural and computer science disciplines.<n>Results reveal that even top-tier models like GPT-4o-image lag behind human performance.
  arXiv  Detail & Related papers  (2025-05-28T08:51:01Z)
• PreGenie: An Agentic Framework for High-quality Visual Presentation Generation [44.93958820783717]
  PreGenie is an agentic and modular framework powered by multimodal large language models (MLLMs) for generating high-quality visual presentations.<n>It operates in two stages: (1) Analysis and Initial Generation, which summarizes multimodal input and generates initial code, and (2) Review and Re-generation, which iteratively reviews intermediate code and rendered slides to produce final, high-quality presentations.
  arXiv  Detail & Related papers  (2025-05-27T18:36:19Z)
• VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Tasks [48.67062958311173]
  VL-GLUE is a multitask benchmark for natural language understanding. We show that this benchmark is quite challenging for existing large-scale vision-language models.
  arXiv  Detail & Related papers  (2024-10-17T15:27:17Z)
• Graph Meets LLMs: Towards Large Graph Models [60.24970313736175]
  We present a perspective paper to discuss the challenges and opportunities associated with developing large graph models. First, we discuss the desired characteristics of large graph models. Then, we present detailed discussions from three key perspectives: representation basis, graph data, and graph models.
  arXiv  Detail & Related papers  (2023-08-28T12:17:51Z)
• Vision-Language Models in Remote Sensing: Current Progress and Future Trends [25.017685538386548]
  Vision-language models enable reasoning about images and their associated textual descriptions, allowing for a deeper understanding of the underlying semantics. Vision-language models can go beyond visual recognition of RS images, model semantic relationships, and generate natural language descriptions of the image. This paper provides a comprehensive review of the research on vision-language models in remote sensing.
  arXiv  Detail & Related papers  (2023-05-09T19:17:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.