BRITE: A Benchmark for Reliable and Interpretable T2V Evaluation on Implausible Scenarios

• URL: http://arxiv.org/abs/2605.00873v1
• Date: Fri, 24 Apr 2026 21:34:52 GMT
• Title: BRITE: A Benchmark for Reliable and Interpretable T2V Evaluation on Implausible Scenarios
• Abstract summary: Existing benchmarks largely overlooked implausible scenarios and do not measure audio-visual alignment.<n>We introduce BRITE, the first framework that unifies implausible prompting, fine-grained assessment of audio-visual consistency, and QA-based interpretable evaluation.
• Score: 4.853153612241331
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid advancement of photorealistic Text-to-Video (T2V) generation brings in an urgent need for up-to-date evaluation methods. Existing benchmarks largely overlooked implausible scenarios and do not measure audio-visual alignment. We introduce BRITE, the first framework that unifies (1) implausible prompting, (2) fine-grained assessment of audio-visual consistency, and (3) QA-based interpretable evaluation into a comprehensive T2V benchmark. Unlike fully automated Multimodal LLM-based pipelines, which are prone to hallucination and prompt ambiguity, BRITE guarantees reliability through a rigorous human-in-the-loop protocol for benchmark creation. Evaluating five state-of-the-art models (Sora 2, Veo 3.1, Runway Gen4.5, Pixverse V5.5, and Qwen3Max), we reveal a critical performance gap: while models excel at static object composition, they exhibit significant degradation in object-action binding and audio-visual synchronization. Our framework offers the community a reliable, interpretable benchmark and evaluation framework that can detect and locate limitations in the next generation of T2V models, especially for off-manifold prompts

Related papers

• MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation [88.7702943548674]
  We introduce MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation.<n>Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios.<n> MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%.
  arXiv  Detail & Related papers  (2026-05-19T17:59:33Z)
• DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models [78.62380562116135]
  We propose DynT2I-Eval, a fully automated dynamic evaluation framework for text-to-image (T2I) models.<n>It constructs a structured visual semantic space from long-form descriptions, decomposing prompts into controllable dimensions.<n>DynT2I-Eval evaluates model performance across text alignment, perceptual quality, and aesthetics.
  arXiv  Detail & Related papers  (2026-05-07T12:53:51Z)
• AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation [42.157453071865056]
  AVGen-Bench is a task-driven benchmark for T2AV generation featuring high-quality prompts across 11 real-world categories.<n>We propose a multi-granular evaluation framework that combines lightweight specialist models with Multimodal Large Language Models (MLLMs)<n>Our evaluation reveals a pronounced gap between strong audio-visual aesthetics and weak semantic reliability, including persistent failures in text rendering, speech coherence, physical reasoning, and a universal breakdown in musical pitch control.
  arXiv  Detail & Related papers  (2026-04-09T17:59:39Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• GEditBench v2: A Human-Aligned Benchmark for General Image Editing [58.86807672117726]
  We introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks.<n>We also propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines.<n> PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average.
  arXiv  Detail & Related papers  (2026-03-30T15:08:32Z)
• MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation [48.84450712826316]
  MSVBench is the first comprehensive benchmark featuring hierarchical scripts and reference images tailored for Multi-Shot Video generation.<n>We propose a hybrid evaluation framework that synergizes the high-level semantic reasoning of Large Multimodal Models with the fine-grained perceptual rigor of domain-specific expert models.
  arXiv  Detail & Related papers  (2026-02-27T12:26:34Z)
• VIPER: Process-aware Evaluation for Generative Video Reasoning [64.86465792516658]
  We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning.<n>Our experiments reveal that state-of-the-art video models achieve only about 20% POC@1.0 and exhibit a significant outcome-hacking.
  arXiv  Detail & Related papers  (2025-12-31T16:31:59Z)
• REVEALER: Reinforcement-Guided Visual Reasoning for Element-Level Text-Image Alignment Evaluation [10.151027538362259]
  REVEALER is a unified framework for element-level alignment evaluation based on reinforcement-guided visual reasoning.<n>Our method enables Multimodal Large Language Models (MLLMs) to explicitly localize semantic elements and derive interpretable alignment judgments.
  arXiv  Detail & Related papers  (2025-12-29T03:24:09Z)
• T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation [41.03487954415606]
  Text-to-Audio-Video (T2AV) generation aims to synthesize temporally coherent video and semantically synchronized audio from natural language.<n>We present T2AV-, a unified benchmark for comprehensive evaluation of T2AV systems.<n>Even the strongest models fall substantially short of human-level realism and cross-modal consistency.
  arXiv  Detail & Related papers  (2025-12-24T10:30:35Z)
• UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation [40.644151228285246]
  We introduce UniGenBench++, a unified semantic assessment benchmark for text-to-image generation.<n>It comprises 600 prompts organized hierarchically to ensure both coverage and efficiency.<n>It provides both English and Chinese versions of each prompt in short and long forms.
  arXiv  Detail & Related papers  (2025-10-21T14:56:46Z)
• UI2V-Bench: An Understanding-based Image-to-video Generation Benchmark [35.157850129371525]
  Image-to-Video (I2V) generation has become a major focus in the field of video synthesis.<n>Existing evaluation benchmarks primarily focus on aspects such as video quality and temporal consistency.<n>We propose UI2V-Bench, a novel benchmark for evaluating I2V models with a focus on semantic understanding and reasoning.
  arXiv  Detail & Related papers  (2025-09-29T08:14:26Z)
• Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models [77.96693360763925]
  Video SimpleQA is the first comprehensive benchmark tailored for factuality evaluation in video contexts.<n>Our work differs from existing video benchmarks through the following key features: Knowledge required: demanding integration of external knowledge beyond the video's explicit narrative.<n>Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance.
  arXiv  Detail & Related papers  (2025-03-24T17:46:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.