LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs

• URL: http://arxiv.org/abs/2603.19217v1
• Date: Thu, 19 Mar 2026 17:58:13 GMT
• Title: LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs
• Abstract summary: This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics.<n>We rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception.<n>Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%.
• Score: 68.35684758116453
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in omnimodal large language models (OmniLLMs) have significantly improved the comprehension of audio and video inputs. However, current evaluations primarily focus on short audio and video clips ranging from 10 seconds to 5 minutes, failing to reflect the demands of real-world applications, where videos typically run for tens of minutes. To address this critical gap, we introduce LVOmniBench, a new benchmark designed specifically for the cross-modal comprehension of long-form audio and video. This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics. Through rigorous manual selection and annotation, LVOmniBench comprises 275 videos, ranging in duration from 10 to 90 minutes, and 1,014 question-answer (QA) pairs. LVOmniBench aims to rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception. Our extensive evaluation reveals that current OmniLLMs encounter significant challenges when processing extended audio-visual inputs. Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%. We anticipate that this dataset, along with our empirical findings, will stimulate further research and the development of advanced models capable of resolving complex cross-modal understanding problems within long-form audio-visual contexts.

Related papers

• AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression [22.818321106501788]
  AVOC is a framework for long-form audio-video understanding in Omni-modal Large Language Models.<n>AVOC achieves state-of-the-art performance on long-form audio-video benchmarks.
  arXiv  Detail & Related papers  (2026-06-23T08:06:58Z)
• VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding [27.284730260765883]
  VideoOdyssey is a benchmark for ultra-long-context and omni-modal video understanding.<n>It spans 11 domains and 54 subcategories with an average video duration of 109 minutes.<n>It provides a comprehensive diagnostic tool to evaluate models across varying context lengths and cognitive loads.
  arXiv  Detail & Related papers  (2026-05-21T18:00:22Z)
• TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos [13.9567665031159]
  Real-world audio-visual understanding requires chaining evidence that is sparse, temporally dispersed, and split across the visual and auditory streams.<n>We introduce TraceAV-Bench, the first benchmark to jointly evaluate multi-hop reasoning over long audio-visual trajectories and multimodal hallucination robustness.
  arXiv  Detail & Related papers  (2026-05-08T11:06:43Z)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos [118.61621763485465]
  MMOU consists of 15,000 carefully curated questions paired with 9038 web-collected videos of varying length.<n>The benchmark covers 13 fundamental skill categories, all of which require integrating evidence across modalities and time.<n>We evaluate 20+ state-of-the-art open-source and proprietary multimodal models on MMOU.
  arXiv  Detail & Related papers  (2026-03-14T22:28:38Z)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs [72.425061028374]
  We introduce OmniVideoBench, a benchmark dedicated to assessing synergistic audio-visual understanding.<n> OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces.<n>We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-12T16:34:00Z)
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs [53.248502396225724]
  AudioMarathon is a benchmark designed to evaluate both understanding and inference efficiency on long-form audio.<n>We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows.<n>The results show large gaps across current LALMs and highlight the need for better temporal reasoning.
  arXiv  Detail & Related papers  (2025-10-08T17:50:16Z)
• MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks [67.31276358668424]
  We introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer.<n> AVHaystacks is an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task.<n>We propose a model-agnostic, multi-agent framework to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystack
  arXiv  Detail & Related papers  (2025-06-08T06:34:29Z)
• SAVEn-Vid: Synergistic Audio-Visual Integration for Enhanced Understanding in Long Video Context [19.224601064352846]
  We introduce SAVEn-Vid, the first-ever long audio-visual video dataset comprising over 58k audio-visual instructions.<n>We present AVBench, a benchmark containing 2,500 QAs designed to evaluate models on enhanced audio-visual comprehension tasks within long video.<n>Experiments demonstrate that SAVEnVideo outperforms the best Video-LLM by 3.61% on the zero-shot long video task (Video-MME) and surpasses the leading audio-visual LLM by 1.29% on the zero-shot audio-visual task (Music-AVQA)
  arXiv  Detail & Related papers  (2024-11-25T09:22:13Z)
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis [120.67048724315619]
  Video-MME is the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis.<n>We extensively evaluate various state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as open-source image models.<n>Our experiments reveal that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models.
  arXiv  Detail & Related papers  (2024-05-31T17:59:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.