Daily Paper Briefing

Abstract Overview

This paper presents a systematic audit of the Physics-IQ benchmark, which is used to evaluate physical understanding in video generative models by comparing generated videos with recordings of real physical experiments. The authors identify three sources of measurement error in the original benchmark: unclear or poorly structured prompts, dataset-level score aggregation that unevenly weights samples, and spurious metric activations caused by artifacts in ground-truth videos. They introduce Physics-IQ Verified, which revises prompts, removes artifact-related activations, and defines a sample-level score that weights samples and metrics equally. The revised benchmark modifies 57.6% of samples, improves more than 34.8% of prompts, and is evaluated on six image-to-video models to study how these changes affect scores and rankings.

Novelty

The main novelty is a benchmark-audit framework that does not propose a new task, but systematically diagnoses and corrects weaknesses in an already influential physical-understanding benchmark. Its distinctive contribution is the combination of prompt revision, artifact cleaning, and a sample-level scoring formulation designed to better isolate physical understanding from confounding factors.

Results

Across six image-to-video models, the verified evaluation produces moderate but meaningful ranking changes relative to the original benchmark, with Kendall’s tau reported as 0.46 and Spearman’s rho as 0.65 between rankings. Improved prompts significantly raise scores for all models except Wan 2.2, while artifact cleaning lowers scores across all models, suggesting that some original scores were inflated by confounds. The new sample-level score generally increases absolute scores but does not materially change model ranking, with its main benefit being finer-grained diagnosis at the sample level.

Key Points

1. Physics-IQ Verified addresses three benchmark issues: prompt quality, score aggregation, and spurious activations in ground-truth videos.
2. The revised benchmark refines 57.6% of samples and improves over 34.8% of prompts, indicating that a substantial portion of the original evaluation protocol required correction.
3. Empirical comparisons on six image-to-video models show that benchmark design choices can alter both absolute scores and comparative rankings, affecting conclusions about model physical understanding.

References

• arXiv: https://arxiv.org/abs/2606.18943v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2606.18943v1
• Hugging Face Papers: https://huggingface.co/papers/2606.18943
• GitHub: https://github.com/google-deepmind/physics-iq-benchmark