FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling

• URL: http://arxiv.org/abs/2608.21839v1
• Date: Sat, 22 Aug 2026 08:21:40 GMT
• Title: FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling
• Abstract summary: We introduce FIRM-Video, a unified checklist-driven data construction framework based on a check-before-score principle.<n>We construct FIRM-Video-90K with 88,044 dimension-specific instances from 29,348 videos, and introduce FIRM-Video-Bench with 750 point-wise human annotations.<n>The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench while consistently delivering the highest VBench Total, Quality, and Semantic Scores.
• Score: 39.98419782895457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reliable reward models are essential for text-to-video evaluation and alignment. However, the trade-off between evaluation accuracy and inference efficiency places high demands on the quality of training supervision. Existing approaches often rely on holistic judges with fixed rubrics or open-ended reasoning, leading to incomplete inspection, unfaithful justification, and entangled attribution. We introduce FIRM-Video, a unified checklist-driven data construction framework based on a check-before-score principle: construct dimension-specific checklists, verify each criterion against temporal visual evidence, and aggregate only verified decisions. For Instruction Following, FIRM-Video decomposes prompts into weighted atomic requirements; for World Coherence, it constructs prompt-calibrated, target-specific checks grounded in visible entities and actions; and for Perceptual Quality, it applies a generic taxonomy of visual defects. The verified criteria and scores are further transformed into natural-language analyses for end-to-end reward modeling. Subsequently, we construct FIRM-Video-90K with 88,044 dimension-specific instances from 29,348 videos, and introduce FIRM-Video-Bench with 750 point-wise human annotations across 250 videos. The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench while consistently delivering the highest VBench Total, Quality, and Semantic Scores in Best-of-8 sampling across three video generators.

Related papers

• VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing [65.49021465098606]
  VideoArgus is a unified framework covering five video generation and editing settings.<n>VideoArgus generates an output-blind, sample-specific rubric once and reuses it to evaluate all corresponding candidate videos.
  arXiv  Detail & Related papers  (2026-08-06T00:28:38Z)
• OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing [53.21868575804758]
  We introduce a comprehensive and structured benchmark for instruction-based video editing (IVE)<n>Our benchmark decomposes editing tasks into multiple video-specific dimensions, including spatial, temporal, audio, and reference-based editing.<n>We propose an evaluation framework that assesses editing quality from four complementary dimensions: accuracy, preservation, realism, and consistency.
  arXiv  Detail & Related papers  (2026-08-05T16:58:48Z)
• NEXT: Reasoning-Driven Video Recommendation via a Vision-Language Model [8.612187834047418]
  Next-interest EXploration Transformer is a reasoning-driven video recommendation framework.<n>Next-8B is a purpose-trained 8B vision-language model with a three-stage recipe.<n>Next shows statistically significant production gains, including +0.53% watch time and +0.51% distinct video exposure.
  arXiv  Detail & Related papers  (2026-06-27T06:51:58Z)
• HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models [28.84235879847985]
  Large vision-language models (LVLMs) have recently shown immense potential in automated content moderation.<n>HarmVideoBench is a diagnostic benchmark comprising 1,379 videos paired with 4,137 multiple-choice questions.<n>We introduce BCR, a benchmark-aligned method that predicts reasoning boundaries and dynamically retrieves context only when needed.
  arXiv  Detail & Related papers  (2026-06-25T15:50:33Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• VideoScore2: Think before You Score in Generative Video Evaluation [69.43069741467603]
  VideoScore2 is a multi-dimensional, interpretable, and human-aligned framework that explicitly evaluates visual quality, text-to-video alignment, and physical/common-sense consistency.<n>Our model is trained on a large-scale dataset VideoFeedback2 containing 27,168 human-annotated videos.
  arXiv  Detail & Related papers  (2025-09-26T18:09:03Z)
• Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Models [51.67019924750931]
  Video-LevelGauge is a benchmark designed to assess positional bias in large video language models (LVLMs)<n>We employ standardized probes and customized contextual setups, allowing flexible control over context length, probe position, and contextual types.<n>Our benchmark comprises 438 manually curated videos spanning multiple types, yielding 1,177 high-quality multiple-choice questions and 120 open-ended questions.
  arXiv  Detail & Related papers  (2025-08-27T07:58:16Z)
• Video-Bench: Human-Aligned Video Generation Benchmark [26.31594706735867]
  Video generation assessment is essential for ensuring that generative models produce visually realistic, high-quality videos.<n>This paper introduces Video-Bench, a comprehensive benchmark featuring a rich prompt suite and extensive evaluation dimensions.<n> Experiments on advanced models including Sora demonstrate that Video-Bench achieves superior alignment with human preferences across all dimensions.
  arXiv  Detail & Related papers  (2025-04-07T10:32:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.