AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs

• URL: http://arxiv.org/abs/2606.07643v1
• Date: Mon, 01 Jun 2026 19:12:09 GMT
• Title: AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs
• Abstract summary: We introduce a benchmark that evaluates Omni-MLLMs across three stages, perception, understanding, and reasoning.<n>AVI-Bench enables fine-grained diagnosis of model capabilities and failure modes.<n>PriSe probes models' primitive audio-visual sensation using unfamiliar, low-semantic stimuli.
• Score: 64.22272455664884
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in Omni-Multimodal Large Language Models (Omni-MLLMs) have enabled strong integration of vision, audio, and language. However, their audio-visual intelligence (AVI) remains insufficiently evaluated due to the lack of systematic and comprehensive benchmarks. We introduce AVI-Bench, a cognitively inspired benchmark that evaluates Omni-MLLMs across three stages, perception, understanding, and reasoning, through cross-modal tasks requiring joint audio-visual interpretation. This design enables fine-grained diagnosis of model capabilities and failure modes. To further assess robustness beyond familiar domains, we propose AVI-Bench-PriSe, an extension that probes models' primitive audio-visual sensation using unfamiliar, low-semantic stimuli, testing generalization beyond common training distributions. Extensive experiments on both open-source and closed-source models reveal substantial limitations in current Omni-MLLMs. Based on these findings, we present a four-level AVI taxonomy. Overall, AVI-Bench provides a principled evaluation framework to guide the development of more robust and generalizable AVI. Project website: https://fudancvl.github.io/AVI-Bench/

Related papers

• Do Audio-Visual Large Language Models Really See and Hear? [66.8531220331223]
  We analyze how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs.<n>Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.
  arXiv  Detail & Related papers  (2026-04-03T00:48:49Z)
• OmniGAIA: Towards Native Omni-Modal AI Agents [103.79729735478924]
  We introduce a benchmark designed to evaluate omni-modal agents on tasks requiring deep reasoning and multi-turn tool execution.<n>We propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception.
  arXiv  Detail & Related papers  (2026-02-26T11:35:04Z)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs [72.425061028374]
  We introduce OmniVideoBench, a benchmark dedicated to assessing synergistic audio-visual understanding.<n> OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces.<n>We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-12T16:34:00Z)
• Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs [72.5567678952768]
  AURELIA is a novel actor-critic based audio-visual (AV) reasoning framework.<n>It distills structured, step-by-step reasoning into AVLLMs at test time.<n>Using AURELIA, we achieve up to a 100% relative improvement, demonstrating its effectiveness.
  arXiv  Detail & Related papers  (2025-03-29T20:42:29Z)
• VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI [17.763461523794806]
  VidEgoThink is a benchmark for evaluating egocentric video understanding capabilities in Embodied AI. We design four key interrelated tasks: video question-answering, hierarchy planning, visual grounding and reward modeling. We conduct extensive experiments with three types of models: API-based MLLMs, open-source image-based MLLMs, and open-source video-based MLLMs.
  arXiv  Detail & Related papers  (2024-10-15T14:08:53Z)
• Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training [48.455597568212944]
  We present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure.<n>In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data.
  arXiv  Detail & Related papers  (2024-10-10T17:59:22Z)
• OmniBench: Towards The Future of Universal Omni-Language Models [63.16606414452612]
  We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously.<n>Our evaluation reveals that open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts.<n>We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance.
  arXiv  Detail & Related papers  (2024-09-23T17:59:05Z)
• B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructions [73.97665608366447]
  Large Vision-Language Models (LVLMs) have shown significant progress in responding well to visual-instructions from users.<n>These instructions, encompassing images and text, are susceptible to both intentional and inadvertent attacks.<n>We introduce B-AVIBench, a framework designed to analyze the robustness of LVLMs when facing various Black-box Adrial Visual-Instructions.
  arXiv  Detail & Related papers  (2024-03-14T12:51:07Z)
• MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria [49.500322937449326]
  Multimodal large language models (MLLMs) have broadened the scope of AI applications. Existing automatic evaluation methodologies for MLLMs are mainly limited in evaluating queries without considering user experiences. We propose a new evaluation paradigm for MLLMs, which is evaluating MLLMs with per-sample criteria using potent MLLM as the judge.
  arXiv  Detail & Related papers  (2023-11-23T12:04:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.