WorldJen: An End-to-End Multi-Dimensional Benchmark for Generative Video Models

• URL: http://arxiv.org/abs/2605.03475v2
• Date: Wed, 06 May 2026 04:45:21 GMT
• Title: WorldJen: An End-to-End Multi-Dimensional Benchmark for Generative Video Models
• Abstract summary: WorldJen is a framework for evaluating generative video models.<n>A blind human preference study is conducted to accumulate 2,696 pairwise annotations from 7 annotators.<n>A VLM-as-a-judge evaluation engine using prompt-specific, dimension-specific Likert questions is developed.
• Score: 0.8640786765448132
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating generative video models remains an open problem. Reference-based metrics such as Structural Similarity Index Measure (SSIM) and Peak Signal to Noise Ratio (PSNR) reward pixel fidelity over semantic correctness, while Frechet Video Distance (FVD) favors distributional textures over physical plausibility. Binary Visual Question Answering (VQA) based benchmarks like VBench~2.0 are prone to yes-bias and rely on low-resolution auditors that miss temporal failures. Moreover, their prompts target a single dimension at a time, multiplying the number of videos required while still not guaranteeing reliable results. WorldJen addresses these limitations directly. Binary VQA is replaced with Likert-scale questionnaires graded by a VLM that receives frames at native video resolution. Video generation costs are addressed by using adversarially curated prompts that are designed to exercise up to 16 quality dimensions simultaneously. The framework is built around two interlocking contributions. First, A blind human preference study is conducted, accumulating (2,696 pairwise annotations from 7 annotators with 100% pair coverage over 50 of the curated prompts $\times$ 6 state-of-the-art video models. A mean inter-annotator agreement of 66.9% is achieved and the study establishes a human ground-truth Bradley-Terry (BT) rating with a three-tier structure. Second, A VLM-as-a-judge evaluation engine using prompt-specific, dimension-specific Likert questionnaires (10 questions per dimension, 47,160 scored responses) judges the videos and reproduces the human-established three-tier BT rating structure independently. The VLM achieves a Spearman $\hatρ=1.000,~p=0.0014$ that is interpreted as tier agreement with the human results. Six focused ablation studies validate the robustness of the VLM evaluation framework. Project page: https://moonmath.ai/worldjen/

Related papers

• VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal [41.67024349583551]
  We introduce VOR Bench, which advances VOR evaluation through three integrated components.<n>First, we present the VOR dataset (VORD), the first benchmark dataset providing both paired edited videos and graffiti masks.<n>Second, we develop rMPAF, a realistic Motion- capable Paired-video Acquisition Framework.<n>Third, we introduce VOR-MDSM, the first perception-driven VLM-based scoring model specifically designed for mask-guided VOR.
  arXiv  Detail & Related papers  (2026-09-15T09:08:00Z)
• BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender [46.08439123011388]
  We test the ability of agents to reconstruct video by asking them to reconstruct realworld videos as animated Blender scenes.<n>The benchmark renders each reconstruction from its animated camera and evaluates it on two axes: (1) Dual VQA measures how many facts the reconstruction preserves, and (2) Latent Similarity measures how closely reconstruction matches source video perceptually.<n>The best model 88.6 Latent Similarity reaches but retains only 53.7% of the source-correcttemporal answers.
  arXiv  Detail & Related papers  (2026-09-14T12:36:05Z)
• TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs [36.2604844032041]
  Video multimodal large language models (MLLMs) can describe what happens in a video, but rarely identify when the supporting evidence occurs.<n>We study generalist video temporal grounding, in which one model predicts a variable-cardinality set of evidence intervals across video lengths, domains, query forms, and viewpoints.<n>TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization.
  arXiv  Detail & Related papers  (2026-07-19T22:04:33Z)
• HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models [28.84235879847985]
  Large vision-language models (LVLMs) have recently shown immense potential in automated content moderation.<n>HarmVideoBench is a diagnostic benchmark comprising 1,379 videos paired with 4,137 multiple-choice questions.<n>We introduce BCR, a benchmark-aligned method that predicts reasoning boundaries and dynamically retrieves context only when needed.
  arXiv  Detail & Related papers  (2026-06-25T15:50:33Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification [73.02304272829785]
  VideoBench is a hierarchical benchmark for long-video answering that rigorously verifies evidence.<n>It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence.<n>Experiments show that even GeminiPro-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting.<n>Results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-04-02T03:29:43Z)
• VideoScore2: Think before You Score in Generative Video Evaluation [69.43069741467603]
  VideoScore2 is a multi-dimensional, interpretable, and human-aligned framework that explicitly evaluates visual quality, text-to-video alignment, and physical/common-sense consistency.<n>Our model is trained on a large-scale dataset VideoFeedback2 containing 27,168 human-annotated videos.
  arXiv  Detail & Related papers  (2025-09-26T18:09:03Z)
• VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models [111.5892290894904]
  VBench is a benchmark suite that dissects "video generation quality" into specific, hierarchical, and disentangled dimensions. We provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception. VBench++ supports evaluating text-to-video and image-to-video.
  arXiv  Detail & Related papers  (2024-11-20T17:54:41Z)
• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models [75.42002690128486]
  TemporalBench is a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. It consists of 10K video question-answer pairs, derived from 2K high-quality human annotations detailing the temporal dynamics in video clips. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Perception Test: A Diagnostic Benchmark for Multimodal Video Models [78.64546291816117]
  We propose a novel multimodal video benchmark to evaluate the perception and reasoning skills of pre-trained multimodal models. The Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  arXiv  Detail & Related papers  (2023-05-23T07:54:37Z)
• A Subjective Quality Study for Video Frame Interpolation [4.151439675744056]
  We describe a subjective quality study for video frame (VFI) based on a newly developed video database, BVI-VFI. BVI-VFI contains 36 reference sequences at three different frame rates and 180 distorted videos generated using five conventional and learning based VFI algorithms.
  arXiv  Detail & Related papers  (2022-02-15T21:13:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.