AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models

• URL: http://arxiv.org/abs/2605.24652v1
• Date: Sat, 23 May 2026 16:42:39 GMT
• Title: AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models
• Abstract summary: We introduce AVBench, a fully automated benchmark tailored for human-centric AV generation.<n> AVBench integrates ten evaluation dimensions designed for human-centered real-world scenarios.<n>It derives continuous evaluation scores from the model's prediction confidence on binary decisions.
• Score: 37.78996308837551
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rapid advances in audio-video (AV) generation have enabled high-fidelity synthesis with synchronized sound, particularly for human-related scenarios involving speech and interactions. Yet evaluation for AV generation remains at an early stage, with only a few coarse-grained benchmarks for human-related scenarios and relying on limited preset evaluations with generic multimodal LLMs, leading to inaccurate assessments of model capabilities. To address these issues, we introduce AVBench, a fully automated benchmark tailored for human-centric AV generation. AVBench is built on two key designs for comprehensive and accurate evaluation: (i) Human-centric and fine-grained metrics. AVBench integrates ten evaluation dimensions designed for human-centered real-world scenarios, covering visual quality, audio quality, and multi-level consistency across modalities. These practical metrics capture human-related details that existing benchmarks often overlook. (ii) Specialized evaluators via preference learning. To address the lack of specialized training data, we construct large-scale supervision by transforming real-world videos into diverse training pairs with controlled perturbations. After fine-tuning on this high-quality dataset, the evaluators learn to reliably detect subtle cross-modal inconsistencies. Crucially, instead of producing discrete textual judgment, AVBench derives continuous evaluation scores from the model's prediction confidence on binary decisions. This probabilistic scoring mechanism enables a more reliable assessment than traditional VQA-style evaluation and aligns closely with human judgment. Taken together, AVBench offers automated evaluation for AV generation, demonstrates strong potential for data filtering, and serves as a differentiable reward signal for Reinforcement Learning from Human Feedback (RLHF).

Related papers

• VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal [41.67024349583551]
  We introduce VOR Bench, which advances VOR evaluation through three integrated components.<n>First, we present the VOR dataset (VORD), the first benchmark dataset providing both paired edited videos and graffiti masks.<n>Second, we develop rMPAF, a realistic Motion- capable Paired-video Acquisition Framework.<n>Third, we introduce VOR-MDSM, the first perception-driven VLM-based scoring model specifically designed for mask-guided VOR.
  arXiv  Detail & Related papers  (2026-09-15T09:08:00Z)
• VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation [79.34043102155964]
  We propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation.<n>We show VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations.
  arXiv  Detail & Related papers  (2026-08-19T06:53:14Z)
• EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation [59.569634087474185]
  We introduce EvalVerse, a comprehensive, pipeline-aware, and expert-calibrated evaluation framework.<n>First, we organize domain knowledge into an evaluation taxonomy aligned with the professional filmmaking workflow.<n>Second, we distill human expert judgments into a curated dataset with large-scale human annotations.<n>Third, we inject this knowledge into Vision-Language Models (VLMs) through an expert-calibrated fine-tuning strategy.
  arXiv  Detail & Related papers  (2026-05-22T06:22:21Z)
• MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation [88.7702943548674]
  We introduce MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation.<n>Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios.<n> MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%.
  arXiv  Detail & Related papers  (2026-05-19T17:59:33Z)
• AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation [42.157453071865056]
  AVGen-Bench is a task-driven benchmark for T2AV generation featuring high-quality prompts across 11 real-world categories.<n>We propose a multi-granular evaluation framework that combines lightweight specialist models with Multimodal Large Language Models (MLLMs)<n>Our evaluation reveals a pronounced gap between strong audio-visual aesthetics and weak semantic reliability, including persistent failures in text rendering, speech coherence, physical reasoning, and a universal breakdown in musical pitch control.
  arXiv  Detail & Related papers  (2026-04-09T17:59:39Z)
• MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation [48.84450712826316]
  MSVBench is the first comprehensive benchmark featuring hierarchical scripts and reference images tailored for Multi-Shot Video generation.<n>We propose a hybrid evaluation framework that synergizes the high-level semantic reasoning of Large Multimodal Models with the fine-grained perceptual rigor of domain-specific expert models.
  arXiv  Detail & Related papers  (2026-02-27T12:26:34Z)
• AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMs [13.950397580491666]
  We introduce AVFakeBench, the first comprehensive audio-video forgery detection benchmark.<n> AVFakeBench comprises 12K carefully curated audio-video questions, covering seven forgery types and four levels of annotations.<n>We evaluate 11 Audio-Video Large Language Models (AV-LMMs) and 2 prevalent detection methods on AVFakeBench.
  arXiv  Detail & Related papers  (2025-11-26T10:33:12Z)
• $C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction [80.57232374640911]
  We propose a model-agnostic strategy called the Mask-And-Recover (MAR)<n>MAR integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules.<n>To better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model.
  arXiv  Detail & Related papers  (2025-04-01T13:01:30Z)
• CONVIQT: Contrastive Video Quality Estimator [63.749184706461826]
  Perceptual video quality assessment (VQA) is an integral component of many streaming and video sharing platforms. Here we consider the problem of learning perceptually relevant video quality representations in a self-supervised manner. Our results indicate that compelling representations with perceptual bearing can be obtained using self-supervised learning.
  arXiv  Detail & Related papers  (2022-06-29T15:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.