SciClaimEval: Cross-modal Claim Verification in Scientific Papers

• URL: http://arxiv.org/abs/2602.07621v2
• Date: Fri, 13 Feb 2026 11:36:02 GMT
• Title: SciClaimEval: Cross-modal Claim Verification in Scientific Papers
• Abstract summary: We present SciClaimEval, a new scientific dataset for the claim verification task.<n>SciClaimEval features authentic claims, including refuted ones, directly extracted from published papers.<n>To create refuted claims, we introduce a novel approach that modifies the supporting evidence (figures and tables), rather than altering the claims or relying on large language models (LLMs) to fabricate contradictions.
• Score: 16.873180911172838
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present SciClaimEval, a new scientific dataset for the claim verification task. Unlike existing resources, SciClaimEval features authentic claims, including refuted ones, directly extracted from published papers. To create refuted claims, we introduce a novel approach that modifies the supporting evidence (figures and tables), rather than altering the claims or relying on large language models (LLMs) to fabricate contradictions. The dataset provides cross-modal evidence with diverse representations: figures are available as images, while tables are provided in multiple formats, including images, LaTeX source, HTML, and JSON. SciClaimEval contains 1,664 annotated samples from 180 papers across three domains, machine learning, natural language processing, and medicine, validated through expert annotation. We benchmark 11 multimodal foundation models, both open-source and proprietary, across the dataset. Results show that figure-based verification remains particularly challenging for all models, as a substantial performance gap remains between the best system and human baseline.

Related papers

• Document-as-Image Representations Fall Short for Scientific Retrieval [54.7382379182364]
  We introduce ArXivDoc, a new benchmark constructed from the underlying sources of scientific papers.<n>We show that document-as-image representations are consistently suboptimal, especially as document length increases.<n>Text-based representations are most effective, even for figure-based queries, by leveraging captions and surrounding context.
  arXiv  Detail & Related papers  (2026-04-20T17:00:17Z)
• DocDjinn: Controllable Synthetic Document Generation with VLMs and Handwriting Diffusion [5.342168661302001]
  We propose a novel framework for controllable synthetic document generation using Vision-Language Models (VLMs)<n>Our approach generates visually plausible and semantically consistent synthetic documents that follow the distribution of an existing source dataset.<n>We show that our framework achieves on average $87%$ of the performance of the full real-world dataset.
  arXiv  Detail & Related papers  (2026-02-25T11:52:13Z)
• Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation [72.34977512403643]
  Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) by retrieving relevant documents from an external corpus.<n>Existing RAG systems primarily focus on unimodal text documents, and often fall short in real-world scenarios where both queries and documents may contain mixed modalities (such as text and images)<n>We propose Nyx, a unified mixed-modal to mixed-modal retriever tailored for Universal Retrieval-Augmented Generation scenarios.
  arXiv  Detail & Related papers  (2025-10-20T09:56:43Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• Towards Knowledge-Aware Document Systems: Modeling Semantic Coverage Relations via Answerability Detection [40.12543056558646]
  We introduce a novel framework for modelling Semantic Coverage Relations (SCR), which classifies document pairs based on how their informational content aligns.<n>We define three core relation types: equivalence, inclusion, and semantic overlap.<n>We use a question answering (QA)-based approach, using the answerability of shared questions across documents as an indicator of semantic coverage.
  arXiv  Detail & Related papers  (2025-09-10T06:00:01Z)
• MCiteBench: A Multimodal Benchmark for Generating Text with Citations [31.793037002996257]
  Multimodal Large Language Models (MLLMs) have advanced in integrating diverse modalities but frequently suffer from hallucination.<n>Existing work primarily focuses on generating citations for text-only content, leaving the challenges of multimodal scenarios largely unexplored.<n>We introduce MCiteBench, the first benchmark designed to assess the ability of MLLMs to generate text with citations in multimodal contexts.
  arXiv  Detail & Related papers  (2025-03-04T13:12:39Z)
• SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents [49.54155332262579]
  We release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations.
  arXiv  Detail & Related papers  (2024-10-28T15:56:49Z)
• LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content [62.816876067499415]
  We propose LiveXiv: a scalable evolving live benchmark based on scientific ArXiv papers.<n>LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs.<n>We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models true abilities.
  arXiv  Detail & Related papers  (2024-10-14T17:51:23Z)
• Bridging Textual and Tabular Worlds for Fact Verification: A Lightweight, Attention-Based Model [34.1224836768324]
  FEVEROUS is a benchmark and research initiative focused on fact extraction and verification tasks. This paper introduces a simple yet powerful model that nullifies the need for modality conversion. Our approach efficiently exploits latent connections between different data types, thereby yielding comprehensive and reliable verdict predictions.
  arXiv  Detail & Related papers  (2024-03-26T03:54:25Z)
• WiCE: Real-World Entailment for Claims in Wikipedia [63.234352061821625]
  We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim. We show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.
  arXiv  Detail & Related papers  (2023-03-02T17:45:32Z)
• End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models [0.0]
  We propose end-to-end multimodal fact-checking and explanation generation. The goal is to assess the truthfulness of a claim by retrieving relevant evidence and predicting a truthfulness label. To support this research, we construct Mocheg, a large-scale dataset consisting of 15,601 claims.
  arXiv  Detail & Related papers  (2022-05-25T04:36:46Z)
• MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization [19.062996443574047]
  We present a new dataset MiRANews and benchmark existing summarization models. We show via data analysis that it's not only the models which are to blame. assisted summarization reduces 55% of hallucinations when compared to single-document summarization models trained on the main article only.
  arXiv  Detail & Related papers  (2021-09-22T10:58:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.