M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency

• URL: http://arxiv.org/abs/2604.01306v1
• Date: Wed, 01 Apr 2026 18:18:10 GMT
• Title: M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency
• Abstract summary: M2-Verify is a large-scale multimodal dataset for checking scientific claim consistency.<n>It provides over 469K instances across 16 domains, rigorously validated through expert audits.
• Score: 17.049806985851994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating scientific arguments requires assessing the strict consistency between a claim and its underlying multimodal evidence. However, existing benchmarks lack the scale, domain diversity, and visual complexity needed to evaluate this alignment realistically. To address this gap, we introduce M2-Verify, a large-scale multimodal dataset for checking scientific claim consistency. Sourced from PubMed and arXiv, M2-Verify provides over 469K instances across 16 domains, rigorously validated through expert audits. Extensive baseline experiments show that state-of-the-art models struggle to maintain robust consistency. While top models achieve up to 85.8\% Micro-F1 on low-complexity medical perturbations, performance drops to 61.6\% on high-complexity challenges like anatomical shifts. Furthermore, expert evaluations expose hallucinations when models generate scientific explanations for their alignment decisions. Finally, we demonstrate our dataset's utility and provide comprehensive usage guidelines.

Related papers

• Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents [56.93800079330285]
  Existing multimodal benchmarks largely evaluate final-answer accuracy.<n>We introduce Sci-MMR, a benchmark for multi-step evidence-grounded scientific reasoning.<n>We find that answer accuracy consistently exceeds complete-evidence recovery rate by more than 20%.
  arXiv  Detail & Related papers  (2026-09-10T08:41:07Z)
• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Finetuning with Scientific Data Increases Hallucinations: A Multi-domain Factuality Evaluation of LLMs [8.700507644617964]
  Large language models (LLMs) are increasingly used to communicate and explain scientific concepts.<n>We evaluate 18 LLMs by comparing each scientifically fine-tuned model against its general-purpose base.
  arXiv  Detail & Related papers  (2026-06-19T12:03:02Z)
• MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents [57.32877731797049]
  MultiHaystack is the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions.<n>We find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus.
  arXiv  Detail & Related papers  (2026-03-05T21:43:02Z)
• Multimodal Fact-Level Attribution for Verifiable Reasoning [80.60864342985748]
  Multimodal large language models (MLLMs) are increasingly used for real-world tasks involving multi-step reasoning and long-form generation.<n>Existing multimodal grounding benchmarks and evaluation methods fail to assess attribution in complex multimodal reasoning.<n>We introduce MuRGAt, a benchmark for evaluating fact-level multimodal attribution in settings that require reasoning beyond direct observation.
  arXiv  Detail & Related papers  (2026-02-12T03:10:02Z)
• PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies [16.537126902822127]
  We introduce PRISMM-Bench, the first benchmark grounded in real reviewer-flagged inconsistencies in scientific papers.<n>We design three tasks, namely inconsistency identification, remedy and pair matching, which assess a model's capacity to detect, correct, and reason over inconsistencies.<n>We benchmark 21 leading LMMs, including large openweight models (GLM-4.5V 106B, InternVL3 78B) and proprietary models (Gemini 2.5 Pro, GPT-5 with high reasoning)
  arXiv  Detail & Related papers  (2025-10-18T13:46:26Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• MuSciClaims: Multimodal Scientific Claim Verification [13.598508835610474]
  We introduce a new benchmark MuSciClaims accompanied by diagnostics tasks.<n>We automatically extract supported claims from scientific articles, which we manually perturb to produce contradicted claims.<n>Our results show most vision-language models are poor (0.3-0.5 F1), with even the best model only achieving 0.72 F1.
  arXiv  Detail & Related papers  (2025-06-05T02:59:51Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs)<n>MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts.<n>It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation [50.061029816288936]
  We present SciFIBench, a scientific figure interpretation benchmark consisting of 2000 questions split between two tasks across 8 categories.<n>The questions are curated from arXiv paper figures and captions, using adversarial filtering to find hard negatives and human verification for quality control.<n>We evaluate 28 LMMs on SciFIBench, finding it to be a challenging benchmark.
  arXiv  Detail & Related papers  (2024-05-14T17:54:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.