VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categories

• URL: http://arxiv.org/abs/2604.10542v1
• Date: Sun, 12 Apr 2026 09:11:55 GMT
• Title: VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categories
• Abstract summary: VidAudio-Bench is a multi-task benchmark for V2A evaluation with four key features.<n>It comprises 1,634 video-text pairs and benchmarks 11 state-of-the-art generation models.<n>It validates all metrics through subjective studies, demonstrating strong consistency with human preferences.
• Score: 47.95995227527911
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video-to-Audio (V2A) generation is essential for immersive multimedia experiences, yet its evaluation remains underexplored. Existing benchmarks typically assess diverse audio types under a unified protocol, overlooking the fine-grained requirements of distinct audio categories. To address this gap, we propose VidAudio-Bench, a multi-task benchmark for V2A evaluation with four key features: (1) Broad Coverage: It encompasses four representative audio categories - sound effects, music, speech, and singing - under both V2A and Video-Text-to-Audio (VT2A) settings. (2) Extensive Evaluation: It comprises 1,634 video-text pairs and benchmarks 11 state-of-the-art generation models. (3) Comprehensive Metrics: It introduces 13 task-specific, reference-free metrics to systematically assess audio quality, video-audio consistency, and text-audio consistency. (4) Human Alignment: It validates all metrics through subjective studies, demonstrating strong consistency with human preferences. Experimental results reveal that current V2A models perform poorly in speech and singing compared to sound effects. Our VT2A results further highlight a fundamental tension between instruction following and visually grounded generation: stronger visual conditioning improves video-audio alignment, but often at the cost of generating the intended audio category. These findings establish VidAudio-Bench as a comprehensive and scalable framework for diagnosing V2A systems and provide new insights into multimodal audio generation.

Related papers

• PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation [23.127789678422698]
  Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality.<n>We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation.
  arXiv  Detail & Related papers  (2026-09-04T08:27:10Z)
• 2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA [47.992210130090065]
  We present ASR-SaSa2VA, a resource-efficient framework for audio-guided video segmentation.<n>Our approach achieves a final score of 80.7 in the 5th PVUW Challenge (MeViS-v2-Audio track), earning the second-place ranking.
  arXiv  Detail & Related papers  (2026-04-27T01:19:56Z)
• AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation [42.157453071865056]
  AVGen-Bench is a task-driven benchmark for T2AV generation featuring high-quality prompts across 11 real-world categories.<n>We propose a multi-granular evaluation framework that combines lightweight specialist models with Multimodal Large Language Models (MLLMs)<n>Our evaluation reveals a pronounced gap between strong audio-visual aesthetics and weak semantic reliability, including persistent failures in text rendering, speech coherence, physical reasoning, and a universal breakdown in musical pitch control.
  arXiv  Detail & Related papers  (2026-04-09T17:59:39Z)
• ALIVE: Animate Your World with Lifelike Audio-Video Generation [50.693986608051716]
  ALIVE is a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation.<n>To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch.<n>ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions.
  arXiv  Detail & Related papers  (2026-02-09T14:06:03Z)
• Omni2Sound: Towards Unified Video-Text-to-Audio Generation [56.11583645408007]
  Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A) and joint video-text-to-audio (VT2A) generation offers significant application flexibility.<n>SoundAtlas is a large-scale dataset (470k pairs) that significantly outperforms existing benchmarks and even human experts in quality.<n>We propose Omni2Sound, a unified VT2A diffusion model supporting flexible input modalities.
  arXiv  Detail & Related papers  (2026-01-06T05:49:41Z)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation [16.067014259345743]
  We evaluate leading vision-only, audio-only, and Omni-LLMs on our dataset.<n>Even the best-performing Omni-LLM achieves an average accuracy of only 62.6%, outperforming uni-modal baselines.
  arXiv  Detail & Related papers  (2025-12-14T17:23:21Z)
• VABench: A Comprehensive Benchmark for Audio-Video Generation [22.00633729850902]
  VABench is a benchmark framework designed to evaluate the capabilities of synchronous audio-video generation.<n>It covers three task types: text-to-audio-video (T2AV), image-to-audio-video (I2AV), and stereo audio-video generation.<n>VABench covers seven major content categories: animals, human sounds, music, environmental sounds, synchronous physical sounds, complex scenes, and virtual worlds.
  arXiv  Detail & Related papers  (2025-12-10T03:57:29Z)
• PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation [57.864929968616586]
  Video-to-Audio (V2A) generation requires balancing four critical perceptual dimensions.<n>We introduce PrismAudio, the first framework to integrate Reinforcement Learning into V2A generation with specialized Chain-of-Thought (CoT) planning.
  arXiv  Detail & Related papers  (2025-11-24T07:11:12Z)
• YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls [10.429203168607147]
  YingSound is a foundation model designed for video-guided sound generation.<n>It supports high-quality audio generation in few-shot settings.<n>We show that YingSound effectively generates high-quality synchronized sounds through automated evaluations and human studies.
  arXiv  Detail & Related papers  (2024-12-12T10:55:57Z)
• Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation [89.96013329530484]
  We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples.
  arXiv  Detail & Related papers  (2023-09-28T13:26:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.