VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing

• URL: http://arxiv.org/abs/2608.05485v1
• Date: Thu, 06 Aug 2026 00:28:38 GMT
• Title: VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing
• Abstract summary: VideoArgus is a unified framework covering five video generation and editing settings.<n>VideoArgus generates an output-blind, sample-specific rubric once and reuses it to evaluate all corresponding candidate videos.
• Score: 65.49021465098606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating generated videos remains challenging because existing benchmarks rely on fixed evaluation content, cover only a subset of generation and editing settings, and provide limited evidence for their scores. We introduce VideoArgus, a unified rubric-grounded framework covering five video generation and editing settings. For each input instance, VideoArgus generates an output-blind, sample-specific rubric once and reuses it to evaluate all corresponding candidate videos. The rubric defines concrete criteria, scoring rules, failure modes, and evidence plans, which guide criterion-specific VLM QA and visual tools to produce evidence-grounded criterion scores, rationales, and a diagnostic report. We further construct VideoArgus-Bench, containing 1,026 curated input instances built from 653 high-quality images and 416 high-quality videos, with all benchmark rubrics pre-generated, frozen, and released. On a separate 1,260-video human-alignment set, VideoArgus achieves higher within-input Spearman and Kendall correlations with human judgments than the corresponding benchmark-specific evaluators across all five tasks. Model rankings also remain largely consistent across different rubric-generation and evaluation-VLM backbones. All code and data are released. Visit our project page: https://zzzmyyzeng.github.io/VideoArgus

Related papers

• FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling [39.98419782895457]
  We introduce FIRM-Video, a unified checklist-driven data construction framework based on a check-before-score principle.<n>We construct FIRM-Video-90K with 88,044 dimension-specific instances from 29,348 videos, and introduce FIRM-Video-Bench with 750 point-wise human annotations.<n>The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench while consistently delivering the highest VBench Total, Quality, and Semantic Scores.
  arXiv  Detail & Related papers  (2026-08-22T08:21:40Z)
• OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing [53.21868575804758]
  We introduce a comprehensive and structured benchmark for instruction-based video editing (IVE)<n>Our benchmark decomposes editing tasks into multiple video-specific dimensions, including spatial, temporal, audio, and reference-based editing.<n>We propose an evaluation framework that assesses editing quality from four complementary dimensions: accuracy, preservation, realism, and consistency.
  arXiv  Detail & Related papers  (2026-08-05T16:58:48Z)
• UniVBench: Towards Unified Evaluation for Video Foundation Models [29.73247324829126]
  Video foundation models aim to integrate video understanding, generation, editing, and instruction following within a single framework.<n>We introduce UniVBench, a benchmark for evaluating video foundation models across four core abilities.<n>Our benchmark substantially expands the complexity of evaluation by incorporating 200 high-quality, diverse and multi-shot videos.
  arXiv  Detail & Related papers  (2026-02-25T12:08:53Z)
• VC-Bench: Pioneering the Video Connecting Benchmark with a Dataset and Evaluation Metrics [83.61875204972465]
  We introduce Video Connecting, a task that aims to generate smooth intermediate video content between given start and end clips.<n>To bridge this gap, we proposed VC-Bench, a novel benchmark specifically designed for video connecting.<n> VC-Bench focuses on three core aspects: Video Quality Score VQS, Start-End Consistency Score SECS, and Transition Smoothness Score TSS.
  arXiv  Detail & Related papers  (2026-01-27T06:15:12Z)
• VideoScore2: Think before You Score in Generative Video Evaluation [69.43069741467603]
  VideoScore2 is a multi-dimensional, interpretable, and human-aligned framework that explicitly evaluates visual quality, text-to-video alignment, and physical/common-sense consistency.<n>Our model is trained on a large-scale dataset VideoFeedback2 containing 27,168 human-annotated videos.
  arXiv  Detail & Related papers  (2025-09-26T18:09:03Z)
• VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation [23.701884816475403]
  Video captions play a crucial role in text-to-video generation tasks.<n>Existing benchmarks inadequately address fine-grained evaluation.<n>We introduce the Fine-grained Video Caption Evaluation Benchmark (VCapsBench)
  arXiv  Detail & Related papers  (2025-05-29T14:34:25Z)
• InstructionBench: An Instructional Video Understanding Benchmark [14.71613140347162]
  We introduce InstructionBench, an instructional video understanding benchmark.<n>We formulate Q&A pairs in open-ended and multiple-choice formats to assess both Coarse-Grained event-level and Fine-Grained object-level reasoning.<n>The benchmark finally contains 5k questions across over 700 videos.
  arXiv  Detail & Related papers  (2025-04-07T13:05:09Z)
• VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models [111.5892290894904]
  VBench is a benchmark suite that dissects "video generation quality" into specific, hierarchical, and disentangled dimensions. We provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception. VBench++ supports evaluating text-to-video and image-to-video.
  arXiv  Detail & Related papers  (2024-11-20T17:54:41Z)
• Beyond Raw Videos: Understanding Edited Videos with Large Multimodal Model [62.38322742493649]
  We build a video VQA benchmark covering editing categories, i.e., effect, funny, meme, and game. Most of the open-source video LMMs perform poorly on the benchmark, indicating a huge domain gap between edited short videos on social media and regular raw videos. To improve the generalization ability of LMMs, we collect a training set for the proposed benchmark based on both Panda-70M/WebVid raw videos and small-scale TikTok/CapCut edited videos.
  arXiv  Detail & Related papers  (2024-06-15T03:28:52Z)
• Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs [20.168429351519055]
  Video understanding is a crucial next step for multimodal large language models (LMLMs)<n>We propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation.<n>We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities.
  arXiv  Detail & Related papers  (2024-06-13T17:50:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.