World Consistency Score: A Unified Metric for Video Generation Quality

• URL: http://arxiv.org/abs/2508.00144v1
• Date: Thu, 31 Jul 2025 20:06:47 GMT
• Title: World Consistency Score: A Unified Metric for Video Generation Quality
• Abstract summary: We introduce World Consistency Score (WCS), a novel unified evaluation metric for generative video models.<n>WCS integrates four interpretable sub-components - object permanence, stability relation, causal compliance, and flicker penalty.<n>We detail the motivation for WCS in the context of existing video evaluation metrics, formalize each submetric, and describe how the weights of the WCS combination are trained.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce World Consistency Score (WCS), a novel unified evaluation metric for generative video models that emphasizes internal world consistency of the generated videos. WCS integrates four interpretable sub-components - object permanence, relation stability, causal compliance, and flicker penalty - each measuring a distinct aspect of temporal and physical coherence in a video. These submetrics are combined via a learned weighted formula to produce a single consistency score that aligns with human judgments. We detail the motivation for WCS in the context of existing video evaluation metrics, formalize each submetric and how it is computed with open-source tools (trackers, action recognizers, CLIP embeddings, optical flow), and describe how the weights of the WCS combination are trained using human preference data. We also outline an experimental validation blueprint: using benchmarks like VBench-2.0, EvalCrafter, and LOVE to test WCS's correlation with human evaluations, performing sensitivity analyses, and comparing WCS against established metrics (FVD, CLIPScore, VBench, FVMD). The proposed WCS offers a comprehensive and interpretable framework for evaluating video generation models on their ability to maintain a coherent "world" over time, addressing gaps left by prior metrics focused only on visual fidelity or prompt alignment.

Related papers

• WorldReward: Reward Modeling for Camera-Conditioned World Models [56.14617966299295]
  We present WorldReward, a VLM-based pairwise preference reward model for camera-conditioned world models.<n>WorldReward decomposes paired videos into action-aligned chunks, organizes each chunk into structured visual evidence, and aggregates chunk-level decisions by voting into separate video-level action and visual-quality preferences.<n>WorldReward achieves the highest agreement on all three dimensions, exceeding GPT-5.5 by 3.42, 1.45, and 3.56 percentage points, respectively.
  arXiv  Detail & Related papers  (2026-09-03T14:53:53Z)
• VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation [79.34043102155964]
  We propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation.<n>We show VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations.
  arXiv  Detail & Related papers  (2026-08-19T06:53:14Z)
• Illusion or Integrity? Geometrical Consistency Metric for AIGC Video Quality Evaluation [8.769587010692774]
  We present a novel benchmark that evaluates the quality of AIGC videos based on their compliance with physical principles.<n>This serves as a proxy for estimating the extent to which generated videos conform to real-world physical rules.<n>We also release a dataset containing 20 scenes across six motion categories.
  arXiv  Detail & Related papers  (2026-08-10T13:28:07Z)
• VC-Bench: Pioneering the Video Connecting Benchmark with a Dataset and Evaluation Metrics [83.61875204972465]
  We introduce Video Connecting, a task that aims to generate smooth intermediate video content between given start and end clips.<n>To bridge this gap, we proposed VC-Bench, a novel benchmark specifically designed for video connecting.<n> VC-Bench focuses on three core aspects: Video Quality Score VQS, Start-End Consistency Score SECS, and Transition Smoothness Score TSS.
  arXiv  Detail & Related papers  (2026-01-27T06:15:12Z)
• Understanding Virality: A Rubric based Vision-Language Model Framework for Short-Form Edutainment Evaluation [8.15791379444665]
  VideoScore-2 does not capture how specific audiovisual attributes drive real audience engagement.<n>We propose a data-driven evaluation framework that uses Vision-Language Models (VLMs) to extract unsupervised audiovisual features.<n>Our approach advances toward robust and explainable video understanding.
  arXiv  Detail & Related papers  (2025-12-24T19:43:59Z)
• VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio [1.0791267046450075]
  VocSim is a training-free benchmark probing the intrinsic geometric alignment of frozen embeddings.<n>VocSim aggregates 125k single-source clips from 19 corpora spanning human speech, animal vocalizations, and environmental sounds.
  arXiv  Detail & Related papers  (2025-12-10T22:13:12Z)
• Q-Save: Towards Scoring and Attribution for Generated Video Evaluation [65.83319736145869]
  We present Q-Save, a new benchmark dataset and model for holistic evaluation of AI-generated video (AIGV) quality.<n>The dataset contains near 10000 videos, each annotated with a scalar mean opinion score (MOS) and fine-grained attribution labels.<n>We propose a unified evaluation model that jointly performs quality scoring and attribution-based explanation.
  arXiv  Detail & Related papers  (2025-11-24T07:00:21Z)
• EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World? [52.99661576320663]
  multimodal large language models (MLLMs) have driven breakthroughs in egocentric vision applications.<n>EOC-Bench is an innovative benchmark designed to systematically evaluate object-centric embodied cognition in dynamic egocentric scenarios.<n>We conduct comprehensive evaluations of various proprietary, open-source, and object-level MLLMs based on EOC-Bench.
  arXiv  Detail & Related papers  (2025-06-05T17:44:12Z)
• Evaluating Text-to-Image Synthesis with a Conditional Fréchet Distance [8.216807467478281]
  evaluating text-to-image synthesis is challenging due to misalignment between established metrics and human preferences.<n>We propose cFreD, a metric that accounts for both visual fidelity and text-prompt alignment.<n>Our findings validate cFreD as a robust, future-proof metric for the systematic evaluation of text-to-image models.
  arXiv  Detail & Related papers  (2025-03-27T17:35:14Z)
• G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o [15.929657348606018]
  We introduce G-VEval, a novel metric inspired by G-Eval and powered by the new GPT-4o.<n>G-VEval uses chain-of-thought reasoning in large multimodal models and supports three modes: reference-free, reference-only, and combined.<n>We also propose MSVD-Eval, a new dataset for video captioning evaluation, to establish a more transparent and consistent framework for both human experts and evaluation metrics.
  arXiv  Detail & Related papers  (2024-12-18T09:23:12Z)
• VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models [111.5892290894904]
  VBench is a benchmark suite that dissects "video generation quality" into specific, hierarchical, and disentangled dimensions. We provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception. VBench++ supports evaluating text-to-video and image-to-video.
  arXiv  Detail & Related papers  (2024-11-20T17:54:41Z)
• FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning [15.363132825156477]
  We introduce FIOVA, a human-centric benchmark tailored for evaluation of large vision-language models (LVLMs)<n>It comprises 3,002 real-world videos (about 33.6s each), each annotated independently by five annotators.<n>We propose FIOVA-DQ, an event-level evaluation metric that incorporates cognitive weights derived from annotator consensus.
  arXiv  Detail & Related papers  (2024-10-20T03:59:54Z)
• PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores [18.26082503192707]
  We develop a PEAVS (Perceptual Evaluation of Audio-Visual Synchrony) score, a novel automatic metric with a 5-point scale that evaluates the quality of audio-visual synchronization. In our experiments, we observe a relative gain 50% over a natural extension of Fr'eche't based metrics for Audio-Visual synchrony.
  arXiv  Detail & Related papers  (2024-04-10T20:32:24Z)
• A Control-Centric Benchmark for Video Prediction [69.22614362800692]
  We propose a benchmark for action-conditioned video prediction in the form of a control benchmark. Our benchmark includes simulated environments with 11 task categories and 310 task instance definitions. We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling.
  arXiv  Detail & Related papers  (2023-04-26T17:59:45Z)
• Video alignment using unsupervised learning of local and global features [0.0]
  We introduce an unsupervised method for alignment that uses global and local features of the frames. In particular, we introduce effective features for each video frame by means of three machine vision tools: person detection, pose estimation, and VGG network. The main advantage of our approach is that no training is required, which makes it applicable for any new type of action without any need to collect training samples for it.
  arXiv  Detail & Related papers  (2023-04-13T22:20:54Z)
• Context Sensing Attention Network for Video-based Person Re-identification [20.865710012336724]
  Video-based person re-identification (ReID) is challenging due to the presence of various interferences in video frames. Recent approaches handle this problem using temporal aggregation strategies. We propose a novel Context Sensing Attention Network (CSA-Net), which improves both the frame feature extraction and temporal aggregation steps.
  arXiv  Detail & Related papers  (2022-07-06T12:48:27Z)
• Group Contextualization for Video Recognition [80.3842253625557]
  Group contextualization (GC) can boost the performance of 2D-CNN (e.g., TSN) and TSM. GC embeds feature with four different kinds of contexts in parallel. Group contextualization can boost the performance of 2D-CNN (e.g., TSN) to a level comparable to the state-the-art video networks.
  arXiv  Detail & Related papers  (2022-03-18T01:49:40Z)
• Local-Global Associative Frame Assemble in Video Re-ID [57.7470971197962]
  Noisy and unrepresentative frames in automatically generated object bounding boxes from video sequences cause challenges in learning discriminative representations in video re-identification (Re-ID) Most existing methods tackle this problem by assessing the importance of video frames according to either their local part alignments or global appearance correlations separately. In this work, we explore jointly both local alignments and global correlations with further consideration of their mutual promotion/reinforcement.
  arXiv  Detail & Related papers  (2021-10-22T19:07:39Z)
• HighlightMe: Detecting Highlights from Human-Centric Videos [52.84233165201391]
  We present a domain- and user-preference-agnostic approach to detect highlightable excerpts from human-centric videos. We use an autoencoder network equipped with spatial-temporal graph convolutions to detect human activities and interactions. We observe a 4-12% improvement in the mean average precision of matching the human-annotated highlights over state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-10-05T01:18:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.