Apple-$π$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
- URL: http://arxiv.org/abs/2607.16401v1
- Date: Fri, 17 Jul 2026 18:00:05 GMT
- Title: Apple-$π$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
- Abstract summary: Apple-PI is the first benchmark that anchors video-model evaluation explicitly in physical laws.<n>Apple-PI comprises three components.<n> Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators.
- Score: 54.58886283991723
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce Apple-PI, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-PI comprises three components. 1) Orchard: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. 2) Benchmark Protocol: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses chain-of-frames prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. 3) Evaluation Suite: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-PI as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.
Related papers
- From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models [42.398664944109676]
We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks.<n>We propose Vid-PRE (Video Reason Prompter and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM.
arXiv Detail & Related papers (2026-09-10T08:40:58Z) - Physics-IQ Verified [58.94877959578469]
We present a systematic audit of the Physics-IQ benchmark and expose shortcomings.<n>We propose three solutions that sharpen how we can measure physical understanding of video generative models.<n>Our resulting benchmark, Physics-IQ Verified, refines 57.6% of all samples and improves over 34.8% of prompts.
arXiv Detail & Related papers (2026-06-17T11:23:52Z) - PhyGround: Benchmarking Physical Reasoning in Generative World Models [46.53245929361594]
We introduce PhyGround, a criteria-grounded benchmark for evaluating physical reasoning in video generation.<n>The benchmark contains 250 curated prompts, each augmented with an expected physical outcome, and a taxonomy of 13 physical laws.<n>We evaluate eight modern video generation models through a large-scale, quality-controlled human study.
arXiv Detail & Related papers (2026-05-11T16:30:51Z) - VIPER: Process-aware Evaluation for Generative Video Reasoning [64.86465792516658]
We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning.<n>Our experiments reveal that state-of-the-art video models achieve only about 20% POC@1.0 and exhibit a significant outcome-hacking.
arXiv Detail & Related papers (2025-12-31T16:31:59Z) - MINERVA: Evaluating Complex Video Reasoning [72.12644008002566]
We provide a new video reasoning dataset called MINERVA for modern multimodal models.<n>Our dataset is multimodal, diverse in terms of video domain and length, and consists of complex multi-step questions.<n>We perform fine-grained error analysis to identify common failure modes across various models, and create a taxonomy of reasoning errors.
arXiv Detail & Related papers (2025-05-01T17:41:49Z) - VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation [66.58048825989239]
VideoPhy-2 is an action-centric dataset for evaluating physical commonsense in generated videos.<n>We perform human evaluation that assesses semantic adherence, physical commonsense, and grounding of physical rules in the generated videos.<n>Our findings reveal major shortcomings, with even the best model achieving only 22% joint performance.
arXiv Detail & Related papers (2025-03-09T22:49:12Z) - How Far is Video Generation from World Model: A Physical Law Perspective [101.24278831609249]
OpenAI's Sora highlights the potential of video generation for developing world models that adhere to physical laws.<n>But the ability of video generation models to discover such laws purely from visual data without human priors can be questioned.<n>In this work, we evaluate across three key scenarios: in-distribution, out-of-distribution, and generalization.
arXiv Detail & Related papers (2024-11-04T18:53:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.