Related papers: Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning

Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning

URL: http://arxiv.org/abs/2605.28192v1
Date: Wed, 27 May 2026 09:11:59 GMT
Title: Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning
Abstract summary: AOP-Agent is an efficient agentic framework built on open-source Omni-LLMs for active omni-modal perception.<n>Experiments on MOV-Bench and OmniVideoBench demonstrate that AOP-Agent consistently improves reasoning performance.
Score: 39.54930573140777
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Multi-hop audio-visual reasoning remains challenging for Omni-LLMs, as relevant evidence is often sparse, temporally dispersed, and distributed across both audio and visual streams. Existing benchmarks provide limited investigation of this setting, typically involving only a limited number of modalities, relevant temporal segments, or reasoning steps. In this work, we introduce MOV-Bench, a benchmark containing 519 carefully curated questions that require multi-hop reasoning over temporally dispersed audio-visual evidence. Evaluations on MOV-Bench reveal that current Omni-LLMs still struggle with multi-hop cross-modal reasoning. To address this challenge, we further propose AOP-Agent, an efficient agentic framework built on open-source Omni-LLMs for active omni-modal perception. By combining a hierarchical omni-modal memory with a collaborative observe-reflect-replan loop, AOP-Agent enables open-source Omni-LLMs to perform active perception without additional training or proprietary models. Experiments on MOV-Bench and OmniVideoBench demonstrate that AOP-Agent consistently improves reasoning performance, with particularly notable gains on long videos and reasoning-intensive questions.

Related papers

Native Active Perception as Reasoning for Omni-Modal Understanding [96.19360756363501]
We propose OmniAgent to formulate video understanding as a POMDP-based iterative Observation-Thought-Action cycle.<n> OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory.<n>We show that OmniAgent achieves state-of-the-art performance among open-source models.
arXiv Detail & Related papers (2026-06-17T17:59:56Z)
LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs [68.35684758116453]
This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics.<n>We rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception.<n>Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%.
arXiv Detail & Related papers (2026-03-19T17:58:13Z)
OmniGAIA: Towards Native Omni-Modal AI Agents [103.79729735478924]
We introduce a benchmark designed to evaluate omni-modal agents on tasks requiring deep reasoning and multi-turn tool execution.<n>We propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception.
arXiv Detail & Related papers (2026-02-26T11:35:04Z)
OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering [12.963349283282447]
We propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning.<n>It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks.<n>It uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries.
arXiv Detail & Related papers (2026-02-03T16:28:24Z)
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding [32.72568710955575]
We present ROMA, a real-time omni-multimodal assistant for unified reactive and proactive interaction.<n> ROMA processes continuous inputs as synchronized multimodal units, aligning dense audio with discrete video frames to handle granularity mismatches.<n>For online decision-making, we introduce a lightweight speak head that decouples response initiation from generation to ensure precise triggering.
arXiv Detail & Related papers (2026-01-15T12:09:04Z)
OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding [23.176694412214157]
We introduce OmniAgent, a fully audio-guided active perception agent.<n>This paper demonstrates a paradigm shift from passive response generation to active multimodal inquiry.
arXiv Detail & Related papers (2025-12-29T17:59:05Z)
OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs [72.425061028374]
We introduce OmniVideoBench, a benchmark dedicated to assessing synergistic audio-visual understanding.<n> OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces.<n>We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.
arXiv Detail & Related papers (2025-10-12T16:34:00Z)
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks [67.31276358668424]
We introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer.<n> AVHaystacks is an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task.<n>We propose a model-agnostic, multi-agent framework to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystack
arXiv Detail & Related papers (2025-06-08T06:34:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.