Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains

• URL: http://arxiv.org/abs/2608.09873v1
• Date: Mon, 10 Aug 2026 17:27:32 GMT
• Title: Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains
• Abstract summary: We introduce Sci-VBench, a benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains.<n>It contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering.
• Score: 15.749662582403614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. It contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering. Each example requires models to generate temporally rich videos that demand scientific reasoning and knowledge-grounded synthesis, going beyond surface-level visual plausibility. We further establish a rubric-based evaluation protocol. Our analysis shows that, under this protocol, both non-expert human evaluators and MLLM-as-Judge systems can achieve relatively high agreement with expert judgments, supporting reproducible evaluation at scale. We benchmark 16 frontier proprietary and open-source models and find that, while automatic perceptual-quality scores cluster tightly across systems, performance on Prompt Grounding and Scientific and Causal Correctness varies substantially, with a pronounced proprietary-open-source gap. These findings show that advances in visual realism have not yet translated into reliable modeling of scientific and causal dynamics.

Related papers

• Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models [49.93401412148884]
  We introduce FEPBench, a benchmark built from carefully selected high-quality scientific illustrations.<n>We evaluate T2I models along three dimensions: instruction faithfulness, reasoning enrichment, and semantic precision.<n>Results show that even state-of-the-art closed-source models, such as GPT Image 2 and Nano Banana Pro, still suffer from text-rendering bottlenecks, limited reasoning enrichment, and difficulty balancing generation richness with precision.
  arXiv  Detail & Related papers  (2026-06-04T09:49:02Z)
• RISE-Video: Can Video Generators Decode Implicit World Rules? [71.92434352963427]
  We present RISE-Video, a pioneering reasoning-oriented benchmark for Text-Image-to-Video (TI2V) synthesis.<n>RISE-Video comprises 467 meticulously human-annotated samples spanning eight rigorous categories.<n>We propose an automated pipeline leveraging Large Multimodal Models (LMMs) to emulate human-centric assessment.
  arXiv  Detail & Related papers  (2026-02-05T18:36:10Z)
• HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery [50.8841471967624]
  HiSciBench is a hierarchical benchmark designed to evaluate foundation models across five levels that mirror the complete scientific workflow.<n>HiSciBench contains 8,735 carefully curated instances spanning six major scientific disciplines.
  arXiv  Detail & Related papers  (2025-12-28T12:08:05Z)
• SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence [99.30934038146965]
  SciEvalKit focuses on the core competencies of scientific intelligence.<n>It supports six major scientific domains, spanning from physics and chemistry to astronomy and materials science.<n>The toolkit is open-sourced and actively maintained to foster community-driven development and progress in AI4Science.
  arXiv  Detail & Related papers  (2025-12-26T17:36:02Z)
• Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench [19.261278000675773]
  VideoScience-Bench is a benchmark designed to evaluate undergraduate-level scientific understanding in video models.<n>The benchmark comprises 200 carefully curated prompts spanning 14 topics and 103 concepts in physics and chemistry.<n>We conduct expert-annotated evaluations across seven state-of-the-art video models in T2V and I2V settings along five dimensions.
  arXiv  Detail & Related papers  (2025-12-02T17:11:23Z)
• SciEducator: Scientific Video Understanding and Educating via Deming-Cycle Multi-Agent System [42.24867393406526]
  SciEducator is a self-evolving multi-agent system for scientific video comprehension and education.<n>Our design reformulates its Plan-Do-Study-Act philosophy into a self-evolving reasoning and feedback mechanism.<n>It can produce multimodal educational content tailored to specific scientific processes.
  arXiv  Detail & Related papers  (2025-11-22T06:54:16Z)
• SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models [89.10286051587151]
  We introduce SciVideoBench, a rigorous benchmark designed to assess advanced video reasoning in scientific contexts.<n>SciVideoBench consists of 1,000 carefully crafted multiple-choice questions derived from cutting-edge scientific experimental videos.<n>Our evaluation highlights significant performance deficits in state-of-the-art proprietary and open-source LMMs.
  arXiv  Detail & Related papers  (2025-10-09T17:59:23Z)
• A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers [251.23085679210206]
  Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research.<n>This survey reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate.<n>We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge.
  arXiv  Detail & Related papers  (2025-08-28T18:30:52Z)
• SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models [36.724471610075696]
  We propose SciHorizon, a comprehensive assessment framework designed to benchmark the readiness of AI4Science.<n>First, we introduce a generalizable framework for assessing AI-ready scientific data, encompassing four key dimensions: Quality, FAIRness, Explainability, and Compliance.<n>We present recommendation lists of AI-ready datasets for Earth, Life, and Materials Sciences, making a novel and original contribution to the field.
  arXiv  Detail & Related papers  (2025-03-12T11:34:41Z)
• SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models [24.394320592613074]
  Large language models (LLMs) are playing an increasingly important role in scientific research.<n>There remains a lack of comprehensive benchmarks to evaluate the breadth and depth of scientific knowledge embedded in these models.<n>We introduce SciKnowEval, a large-scale dataset designed to assess LLMs across five progressive levels of scientific understanding.
  arXiv  Detail & Related papers  (2024-06-13T13:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.