VGI-BENCH: Probing Visual Intelligence in Video Generation Models
Abstract Overview
This paper introduces VGI-Bench, a benchmark for evaluating visual intelligence in video generation models through process-sensitive, visually grounded tasks. The benchmark contains 27 tasks and 810 instances organized under a two-level taxonomy of four task domains and seven skill tags, featuring photorealistic-style inputs and calibrated difficulty near current capability boundaries. The evaluation framework combines global goal completion and local process validity to verify whether models achieve goals through valid intermediate trajectories. Experiments across commercial and open-source models show that while current video generators exhibit emerging visual reasoning capabilities, they remain far from reliable.
Novelty
VGI-Bench introduces a visual reasoning benchmark for video generation models characterized by realistic inputs, process-sensitive tasks that require valid state rollouts, and calibrated task feasibility. It provides a specialized dual-metric evaluation scheme scoring both completion and rule-preserving process validity, alongside diagnostics covering failure modes, input condition sensitivity, synthetic fine-tuning transfer, and internal denoising trajectory dynamics.
Results
Commercial models lead open-source models, yet overall performance is modest, with the top model, Seedance 2.0, scoring 51.0 overall. Structured Puzzles is the most challenging domain, while Topology and Temporal reasoning represent the weakest skill dimensions. Diagnostic findings show that synthetic fine-tuning yields uneven transfer bounded by structural task overlap, and denoising trajectories exhibit minimal self-correction (under 1%), primarily refining early hypotheses.
Key Points
- VGI-Bench benchmarks video generation models across 27 process-sensitive tasks and 810 instances structured by four domains and seven cognitive skill tags.
- The evaluation requires both overall goal completion and rule-preserving intermediate trajectory validity, using adaptive frame sampling and sliding focus windows.
- Empirical results show limited reasoning reliability across current models, with recurring failures in physical plausibility, rule adherence, temporal consistency, and dynamic self-correction during denoising.