VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editing

• URL: http://arxiv.org/abs/2607.22632v1
• Date: Fri, 19 Jun 2026 17:10:52 GMT
• Title: VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editing
• Abstract summary: We define a comprehensive vlog evaluation framework guided by professional vlog creators and product managers.<n>We then curate a large-scale dataset of 100k vlog edits and a dedicated benchmark, VRMBench, to evaluate the vlog rewarding capabilities of Multimodal Large Language Models.<n>Finally, we present VlogReward, a robust vlog reward model that can provide both fine-grained multi-dimensional scores and actionable feedback for iterative refinement.
• Score: 32.06471299558917
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The rapid rise of vlogs as a personalized storytelling medium has created a demand for automated systems to evaluate and refine vlog editing plans. However, vlog assessment is highly subjective and remains challenging due to a lack of standardized criteria, dataset and benchmark, and effective reward models. To address these challenges, we define a comprehensive vlog evaluation framework guided by professional vlog creators and product managers, establishing a taxonomy of six key dimensions, i.e., Creativity, Consistency, Concept Design, Cinematography, Narration, and Pacing. Subsequently, we curate a large-scale dataset of 100k vlog edits and a dedicated benchmark, VRMBench, to evaluate the vlog rewarding capabilities of Multimodal Large Language Models (MLLMs). Finally, we present VlogReward, a robust vlog reward model that can provide both fine-grained multi-dimensional scores and actionable feedback for iterative refinement. Technically, we enhance the Group Relative Policy Optimization (GRPO) framework by introducing an adjustable inter-group comparison reward, which mitigates the "direction blindness" issue of standard GRPO and enables the model to better distinguish varied-quality edits. VlogReward achieves state-of-the-art results that significantly outperform existing MLLMs, including GPT-5 and Gemini-3-Pro. We hope that our study can help vlog creators and foster automated vlog evaluation and refinement systems.

Related papers

• Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation [73.85089704122315]
  We propose an iterative refinement procedure that discovers a per-VLM evaluation taxonomy.<n>We calibrate the VLM's per-dimension scores to human physical-commonsense ratings, diagnose which dimensions it scores unreliably or redundantly, and prompt an LLM to repair them.<n>The refined taxonomy outperforms the global-schema baseline on held-out videos for every VLM tested, with a mean relative improvement of approximately 32%.
  arXiv  Detail & Related papers  (2026-06-22T06:58:39Z)
• VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation [65.0648741395158]
  VGA-Bench is a unified benchmark for evaluation of video generation quality and aesthetic quality.<n>We design 1,016 diverse prompts and generate a large-scale dataset of over 60,000 videos using 12 video generation models.<n>Our models achieve reliable alignment with human judgments, offering both accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-04-11T09:44:39Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• Analytic Score Optimization for Multi Dimension Video Quality Assessment [14.857118087904206]
  We present a large-scale multi-dimensional VQA dataset that encompasses diverse User-Generated Content(UGC) annotated across five key quality dimensions.<n>Each video in our dataset is scored by over 3 human raters on these dimensions, with fine-grained sub-attribute labels.<n>We introduce Analytic Score Optimization (ASO), a theoretically grounded post-training objective derived for multi-dimensional VQA.
  arXiv  Detail & Related papers  (2026-02-18T20:34:24Z)
• Q-Save: Towards Scoring and Attribution for Generated Video Evaluation [65.83319736145869]
  We present Q-Save, a new benchmark dataset and model for holistic evaluation of AI-generated video (AIGV) quality.<n>The dataset contains near 10000 videos, each annotated with a scalar mean opinion score (MOS) and fine-grained attribution labels.<n>We propose a unified evaluation model that jointly performs quality scoring and attribution-based explanation.
  arXiv  Detail & Related papers  (2025-11-24T07:00:21Z)
• CAMP-VQA: Caption-Embedded Multimodal Perception for No-Reference Quality Assessment of Compressed Video [9.172799792564009]
  We propose CAMP-VQA, a novel NR-VQA framework that exploits the semantic understanding capabilities of large models.<n>Our approach introduces a quality-aware video metadata mechanism that integrates key fragments extracted from inter-frame variations.<n>Our model consistently outperforms existing NR-VQA methods, achieving improved accuracy without the need for costly manual fine-grained annotations.
  arXiv  Detail & Related papers  (2025-11-10T16:37:47Z)
• Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation [78.24409139109507]
  We introduce Stable Cinemetrics, a framework that formalizes filmmaking controls into four disentangled, hierarchical control nodes.<n>We conduct a large-scale human study spanning 10+ models and 20K videos, annotated by a pool of 80+ film professionals.<n>SCINE is the first approach to situate professional video generation within the landscape of video generative models.
  arXiv  Detail & Related papers  (2025-09-30T17:22:18Z)
• PersonaVlog: Personalized Multimodal Vlog Generation with Multi-Agent Collaboration and Iterative Self-Correction [16.08952891878684]
  PersonaVlog is an automated multimodal stylized Vlog generation framework.<n>We propose a multi-agent collaboration framework based on Multimodal Large Language Models (MLLMs)<n>We also propose ThemeVlogEval, a theme-based automated benchmarking framework.
  arXiv  Detail & Related papers  (2025-08-19T08:10:43Z)
• Value-Spectrum: Quantifying Preferences of Vision-Language Models via Value Decomposition in Social Media Contexts [33.12056808870413]
  We introduce Value-Spectrum, a novel Visual Question Answering (VQA) benchmark aimed at assessing Vision-Language Models (VLMs)<n>We design a VLM agent pipeline to simulate video browsing and construct a vector database comprising over 50,000 short videos from TikTok, YouTube Shorts, and Instagram Reels.<n>These videos span multiple months and cover diverse topics, including family, health, hobbies, society, technology, etc.
  arXiv  Detail & Related papers  (2024-11-18T11:31:10Z)
• Vlogger: Make Your Dream A Vlog [67.50445251570173]
  Vlogger is a generic AI system for generating a minute-level video blog (i.e., vlog) of user descriptions. We invoke various foundation models to play the critical roles of vlog professionals, including Script, (2) Actor, (3) ShowMaker, and (4) Voicer. Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor.
  arXiv  Detail & Related papers  (2024-01-17T18:55:12Z)
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models [70.19437817951673]
  We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics.
  arXiv  Detail & Related papers  (2023-10-17T17:50:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.