SIQA: Toward Reliable Scientific Image Quality Assessment

• URL: http://arxiv.org/abs/2603.06700v1
• Date: Thu, 05 Mar 2026 06:57:26 GMT
• Title: SIQA: Toward Reliable Scientific Image Quality Assessment
• Abstract summary: We introduce Scientific Image Quality Assessment (SIQA), a framework that models scientific image quality along two complementary dimensions.<n>We design two evaluation protocols: SIQA-U (Understanding), which measures semantic comprehension of scientific content through multiple-choice tasks, and SIQA-S (Scoring), which evaluates alignment with expert quality judgments.<n>Experiments across representative multimodal large language models (MLLMs) reveal a consistent discrepancy between scoring alignment and scientific understanding.
• Score: 72.41803245808924
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scientific images fundamentally differ from natural and AI-generated images in that they encode structured domain knowledge rather than merely depict visual scenes. Assessing their quality therefore requires evaluating not only perceptual fidelity but also scientific correctness and logical completeness. However, existing image quality assessment (IQA) paradigms primarily focus on perceptual distortions or image-text alignment, implicitly assuming that depicted content is factually valid. This assumption breaks down in scientific contexts, where visually plausible figures may still contain conceptual errors or incomplete reasoning. To address this gap, we introduce Scientific Image Quality Assessment (SIQA), a framework that models scientific image quality along two complementary dimensions: Knowledge (Scientific Validity and Scientific Completeness) and Perception (Cognitive Clarity and Disciplinary Conformity). To operationalize this formulation, we design two evaluation protocols: SIQA-U (Understanding), which measures semantic comprehension of scientific content through multiple-choice tasks, and SIQA-S (Scoring), which evaluates alignment with expert quality judgments. We further construct the SIQA Challenge, consisting of an expert-annotated benchmark and a large-scale training set. Experiments across representative multimodal large language models (MLLMs) reveal a consistent discrepancy between scoring alignment and scientific understanding. While models can achieve strong agreement with expert ratings under SIQA-S, their performance on SIQA-U remains substantially lower. Fine-tuning improves both metrics, yet gains in scoring consistently outpace improvements in understanding. These results suggest that rating consistency alone may not reliably reflect scientific comprehension, underscoring the necessity of multidimensional evaluation for scientific image quality assessment.

Related papers

• SciQNet: Two-Stage Multimodal Adaptation for Scientific Image Quality Assessment [0.41998444721319217]
  SciQNet is a two-stage multimodal adaptation framework for scientific image quality assessment.<n>The first stage performs domain-adaptive pretraining on scientific document images.<n>The second stage conducts task-specific fine-tuning with joint scoring and understanding supervision.
  arXiv  Detail & Related papers  (2026-08-06T07:33:23Z)
• SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context [27.236953887247125]
  We propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions.<n>The data covers top computer-science conferences from 2020 to 2025.<n>Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context.
  arXiv  Detail & Related papers  (2026-07-29T16:07:44Z)
• Towards Characterizing Scientific Image Utility and Upgradability [71.77263034120624]
  We propose a framework for scientific image evaluation.<n>We categorize scientific image corruption into four fundamental types: Detail Distortion, Incompleteness, False Content, and Entity Confusion.<n>The framework implements a two-stage evaluation protocol: the textitUtility stage evaluates error detection capability and repair instruction generation, while the textitUpgradability stage assesses whether corrections faithfully restore scientific validity without compromising existing accurate information.
  arXiv  Detail & Related papers  (2026-06-02T09:42:29Z)
• StableI2I: Spotting Unintended Changes in Image-to-Image Transition [35.72064037549274]
  We propose StableI2I, a unified and dynamic evaluation framework that measures content fidelity and pre-post consistency.<n>Our framework serves as a practical and reliable evaluation tool for diagnosing content consistency and benchmarking model performance in real-world I2I systems.
  arXiv  Detail & Related papers  (2026-05-06T03:28:05Z)
• Q-DeepSight: Incentivizing Thinking with Images for Image Quality Assessment and Refinement [58.15004031934379]
  We propose Q-DeepSight, a think-with-image framework that emulates this human-like process.<n>Q-DeepSight achieves state-of-the-art performance across diverse benchmarks, including natural, restored, and AI-generated content.<n>We demonstrate its practical value with Perceptual-in-Generation (PiG), a training-free framework where Q-DeepSight's diagnoses guide iterative image enhancement.
  arXiv  Detail & Related papers  (2026-04-18T06:10:57Z)
• SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence [60.202862987441684]
  We introduce scientific instruction following: the capability to solve problems while strictly adhering to the constraints that establish scientific validity.<n>Specifically, we introduce SciIF, a multi-discipline benchmark that evaluates this capability by pairing university-level problems with a fixed catalog of constraints.<n>By measuring both solution correctness and multi-constraint adherence, SciIF enables finegrained diagnosis of compositional reasoning failures.
  arXiv  Detail & Related papers  (2026-01-08T09:45:58Z)
• ChatGPT and Gemini participated in the Korean College Scholastic Ability Test -- Earth Science I [0.0]
  This study utilizes the Earth Science I section of the 2025 Korean College Scholastic Ability Test (CSAT) to analyze the multimodal scientific reasoning capabilities and cognitive limitations of state-of-the-art Large Language Models (LLMs)<n> Quantitative results indicated that unstructured inputs led to significant performance degradation due to segmentation and Optical Character Recognition (OCR) failures.<n>By exploiting AI's weaknesses, educators can distinguish genuine student competency from AI-generated responses, thereby ensuring assessment fairness.
  arXiv  Detail & Related papers  (2025-12-17T10:46:41Z)
• Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content [71.46991494014382]
  We introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images.<n>Q-Real consists of 3,088 images generated by popular text-to-image models.<n>We construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning.
  arXiv  Detail & Related papers  (2025-11-21T02:43:17Z)
• Teaching LMMs for Image Quality Scoring and Interpreting [71.1335005098584]
  We propose Q-SiT (Quality Scoring and Interpreting joint Teaching), a unified framework that enables image quality scoring and interpreting simultaneously.<n>Q-SiT is the first model capable of simultaneously performing image quality scoring and interpreting tasks, along with its lightweight variant, Q-SiT-mini.<n> Experimental results demonstrate that Q-SiT achieves strong performance in both tasks with superior generalization IQA abilities.
  arXiv  Detail & Related papers  (2025-03-12T09:39:33Z)
• Image Intrinsic Scale Assessment: Bridging the Gap Between Quality and Resolution [4.896425819316727]
  We introduce the Image Intrinsic Scale (IIS), defined as the largest scale where an image exhibits its highest perceived quality.<n>We present the Image Intrinsic Scale Assessment (IISA) task, which involves subjectively measuring and predicting the IIS based on human judgments.
  arXiv  Detail & Related papers  (2025-02-10T13:54:55Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Pairwise Comparisons Are All You Need [22.798716660911833]
  Blind image quality assessment (BIQA) approaches often fall short in real-world scenarios due to their reliance on a generic quality standard applied uniformly across diverse images. This paper introduces PICNIQ, a pairwise comparison framework designed to bypass the limitations of conventional BIQA. By employing psychometric scaling algorithms, PICNIQ transforms pairwise comparisons into just-objectionable-difference (JOD) quality scores, offering a granular and interpretable measure of image quality.
  arXiv  Detail & Related papers  (2024-03-13T23:43:36Z)
• Cuid: A new study of perceived image quality and its subjective assessment [30.698984450985318]
  We present a new study of image quality perception where subjective ratings were collected in a controlled lab environment. We investigate how quality perception is affected by a combination of different categories of images and different types and levels of distortions.
  arXiv  Detail & Related papers  (2020-09-28T13:14:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.