SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

• URL: http://arxiv.org/abs/2607.27084v1
• Date: Wed, 29 Jul 2026 16:07:44 GMT
• Title: SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context
• Abstract summary: We propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions.<n>The data covers top computer-science conferences from 2020 to 2025.<n>Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context.
• Score: 27.236953887247125
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) methods are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visual-surface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness. To address this, we propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers top computer-science conferences from 2020 to 2025; 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into gold-standard annotations. Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context. To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation framework SFQ-Agent to achieve auditable and refined scoring through the collection and fusion of modal evidence. Multiple mainstream large models were evaluated on the test subset eval1200, and SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest consistency rate (93.4%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes.

Related papers

• SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence [7.2640055022009635]
  We propose SciFigAlign, a fine-tuned multimodal scorer that grounds figure quality assessment in manuscript evidence.<n>Under paper-level splits, SciFigAlign achieves a macro MAE of 0.3524 and a within-paper pairwise accuracy of 81.64% on the test set.
  arXiv  Detail & Related papers  (2026-07-29T15:54:14Z)
• SIQA: Toward Reliable Scientific Image Quality Assessment [72.41803245808924]
  We introduce Scientific Image Quality Assessment (SIQA), a framework that models scientific image quality along two complementary dimensions.<n>We design two evaluation protocols: SIQA-U (Understanding), which measures semantic comprehension of scientific content through multiple-choice tasks, and SIQA-S (Scoring), which evaluates alignment with expert quality judgments.<n>Experiments across representative multimodal large language models (MLLMs) reveal a consistent discrepancy between scoring alignment and scientific understanding.
  arXiv  Detail & Related papers  (2026-03-05T06:57:26Z)
• RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension [65.81339691942757]
  RPC-Bench is a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers.<n>We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts.
  arXiv  Detail & Related papers  (2026-01-14T11:37:00Z)
• ProImage-Bench: Rubric-Based Evaluation for Professional Image Generation [151.75112778479468]
  We study professional image generation, where a model must synthesize scientifically precise illustrations from technical descriptions.<n>For 654 figures collected from real textbooks and technical reports, we construct detailed image instructions and a hierarchy of rubrics that decompose correctness into 6,076 criteria and 44,131 binary checks.<n>We benchmark several representative text-to-image models on ProImage-Bench and find that, despite strong open-domain performance, the best base model reaches only 0.791 accuracy and 0.553 criterion score overall.
  arXiv  Detail & Related papers  (2025-12-13T07:13:43Z)
• VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations [40.10675156646689]
  multimodal large language models (MLLMs) have shown promising performance in aesthetic assessment of natural images.<n>No systematic benchmark exists for measuring their capabilities in evaluating visualizations.<n>We propose VisJudge, a model specifically designed for visualization aesthetics and quality assessment.
  arXiv  Detail & Related papers  (2025-10-25T17:31:02Z)
• SciArena: An Open Evaluation Platform for Foundation Models in Scientific Literature Tasks [87.29946641069068]
  We present SciArena, an open and collaborative platform for evaluating foundation models on scientific literature tasks.<n>By leveraging collective intelligence, SciArena offers a community-driven evaluation of model performance on open-ended scientific tasks.<n>We release SciArena-Eval, a meta-evaluation benchmark based on our collected preference data.
  arXiv  Detail & Related papers  (2025-07-01T17:51:59Z)
• Evaluating Text-to-Image and Text-to-Video Synthesis with a Conditional Fréchet Distance [13.599366291595672]
  We propose cFreD, a metric that unifies the assessment of visual fidelity and text-prompt consistency into a single score.<n>Our findings validate cFreD as a robust, future-proof metric for the systematic evaluation of text conditioned models.
  arXiv  Detail & Related papers  (2025-03-27T17:35:14Z)
• SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval [64.03631654052445]
  Current benchmarks for evaluating MMIR performance in image-text pairing within the scientific domain show a notable gap. We develop a specialised scientific MMIR benchmark by leveraging open-access paper collections. This benchmark comprises 530K meticulously curated image-text pairs, extracted from figures and tables with detailed captions in scientific documents.
  arXiv  Detail & Related papers  (2024-01-24T14:23:12Z)
• Holistic Evaluation of Text-To-Image Models [153.47415461488097]
  We introduce a new benchmark, Holistic Evaluation of Text-to-Image Models (HEIM) We identify 12 aspects, including text-image alignment, image quality, aesthetics, originality, reasoning, knowledge, bias, toxicity, fairness, robustness, multilinguality, and efficiency. Our results reveal that no single model excels in all aspects, with different models demonstrating different strengths.
  arXiv  Detail & Related papers  (2023-11-07T19:00:56Z)
• GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions [22.181665641802468]
  This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions. We first constructed SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding.
  arXiv  Detail & Related papers  (2023-10-23T23:24:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.