AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation

• URL: http://arxiv.org/abs/2604.08540v1
• Date: Thu, 09 Apr 2026 17:59:39 GMT
• Title: AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation
• Abstract summary: AVGen-Bench is a task-driven benchmark for T2AV generation featuring high-quality prompts across 11 real-world categories.<n>We propose a multi-granular evaluation framework that combines lightweight specialist models with Multimodal Large Language Models (MLLMs)<n>Our evaluation reveals a pronounced gap between strong audio-visual aesthetics and weak semantic reliability, including persistent failures in text rendering, speech coherence, physical reasoning, and a universal breakdown in musical pitch control.
• Score: 42.157453071865056
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-Audio-Video (T2AV) generation is rapidly becoming a core interface for media creation, yet its evaluation remains fragmented. Existing benchmarks largely assess audio and video in isolation or rely on coarse embedding similarity, failing to capture the fine-grained joint correctness required by realistic prompts. We introduce AVGen-Bench, a task-driven benchmark for T2AV generation featuring high-quality prompts across 11 real-world categories. To support comprehensive assessment, we propose a multi-granular evaluation framework that combines lightweight specialist models with Multimodal Large Language Models (MLLMs), enabling evaluation from perceptual quality to fine-grained semantic controllability. Our evaluation reveals a pronounced gap between strong audio-visual aesthetics and weak semantic reliability, including persistent failures in text rendering, speech coherence, physical reasoning, and a universal breakdown in musical pitch control. Code and benchmark resources are available at http://aka.ms/avgenbench.

Related papers

• PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation [23.127789678422698]
  Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality.<n>We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation.
  arXiv  Detail & Related papers  (2026-09-04T08:27:10Z)
• MMAE: A Massive Multitask Audio Editing Benchmark [66.74229858407413]
  MMAE is the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing.<n>It comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework.<n>Our evaluation reveals that current systems remain far from achieving reliable edits.
  arXiv  Detail & Related papers  (2026-06-05T12:52:41Z)
• MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation [56.428467142804756]
  We introduce MTAVG-Bench 2.0, a benchmark for diagnosing failure modes of cinematic expressiveness in multi-talker audio-video generation.<n>MTAVG-Bench 2.0 targets short-drama and scene-level generation, and establishes a high-level failure taxonomy spanning acting, narrative, atmosphere, and audio-visual language.<n> Experimental results show that commercial omni models such as Gemini substantially outperform other evaluators.
  arXiv  Detail & Related papers  (2026-05-27T06:38:54Z)
• LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV [39.241237759347825]
  LongAV- is a systematic benchmark for minute-long audio-visual generation.<n>It consists of 284 test cases spanning text-to-audio-video (T2AV), image-to---video (I2AV), and video-to-audio-video (V2AV)<n>The framework evaluates more than 20 fine-grained dimensions covering within-audio-segment quality, cross-segment consistency, global narrative coherence, semantic alignment, and audio-visual alignment.
  arXiv  Detail & Related papers  (2026-05-25T18:12:09Z)
• MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation [88.7702943548674]
  We introduce MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation.<n>Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios.<n> MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%.
  arXiv  Detail & Related papers  (2026-05-19T17:59:33Z)
• Omni2Sound: Towards Unified Video-Text-to-Audio Generation [56.11583645408007]
  Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A) and joint video-text-to-audio (VT2A) generation offers significant application flexibility.<n>SoundAtlas is a large-scale dataset (470k pairs) that significantly outperforms existing benchmarks and even human experts in quality.<n>We propose Omni2Sound, a unified VT2A diffusion model supporting flexible input modalities.
  arXiv  Detail & Related papers  (2026-01-06T05:49:41Z)
• T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation [41.03487954415606]
  Text-to-Audio-Video (T2AV) generation aims to synthesize temporally coherent video and semantically synchronized audio from natural language.<n>We present T2AV-, a unified benchmark for comprehensive evaluation of T2AV systems.<n>Even the strongest models fall substantially short of human-level realism and cross-modal consistency.
  arXiv  Detail & Related papers  (2025-12-24T10:30:35Z)
• AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMs [13.950397580491666]
  We introduce AVFakeBench, the first comprehensive audio-video forgery detection benchmark.<n> AVFakeBench comprises 12K carefully curated audio-video questions, covering seven forgery types and four levels of annotations.<n>We evaluate 11 Audio-Video Large Language Models (AV-LMMs) and 2 prevalent detection methods on AVFakeBench.
  arXiv  Detail & Related papers  (2025-11-26T10:33:12Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks [67.31276358668424]
  We introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer.<n> AVHaystacks is an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task.<n>We propose a model-agnostic, multi-agent framework to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystack
  arXiv  Detail & Related papers  (2025-06-08T06:34:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.