OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering

• URL: http://arxiv.org/abs/2602.03707v2
• Date: Wed, 04 Feb 2026 03:33:14 GMT
• Title: OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering
• Abstract summary: We propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning.<n>It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks.<n>It uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries.
• Score: 12.963349283282447
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Long-horizon omnimodal question answering answers questions by reasoning over text, images, audio, and video. Despite recent progress on OmniLLMs, low-resource long audio-video QA still suffers from costly dense encoding, weak fine-grained retrieval, limited proactive planning, and no clear end-to-end optimization.To address these issues, we propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning. It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks. Moreover, it uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries. Furthermore, we apply group relative policy optimization to jointly improve tool use and answer quality over time. Experiments on OmniVideoBench, WorldSense, and Daily-Omni show that OmniRAG-Agent consistently outperforms prior methods under low-resource settings and achieves strong results, with ablations validating each component.

Related papers

• OmniReasoner: Thinking with Long Audio-Video via Native Tool Use [25.032571988845262]
  We introduce OmniReasoner, a tool-use framework for thinking with long audio-video.<n>It learns to decide whether and where to call a zoom-in tool before answering.<n>TimeAnchor keeps the tool's temporal argument valid and round-trip-consistent across granularities.
  arXiv  Detail & Related papers  (2026-07-21T17:57:50Z)
• AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression [22.818321106501788]
  AVOC is a framework for long-form audio-video understanding in Omni-modal Large Language Models.<n>AVOC achieves state-of-the-art performance on long-form audio-video benchmarks.
  arXiv  Detail & Related papers  (2026-06-23T08:06:58Z)
• Native Active Perception as Reasoning for Omni-Modal Understanding [96.19360756363501]
  We propose OmniAgent to formulate video understanding as a POMDP-based iterative Observation-Thought-Action cycle.<n> OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory.<n>We show that OmniAgent achieves state-of-the-art performance among open-source models.
  arXiv  Detail & Related papers  (2026-06-17T17:59:56Z)
• Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning [39.54930573140777]
  AOP-Agent is an efficient agentic framework built on open-source Omni-LLMs for active omni-modal perception.<n>Experiments on MOV-Bench and OmniVideoBench demonstrate that AOP-Agent consistently improves reasoning performance.
  arXiv  Detail & Related papers  (2026-05-27T09:11:59Z)
• LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs [68.35684758116453]
  This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics.<n>We rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception.<n>Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%.
  arXiv  Detail & Related papers  (2026-03-19T17:58:13Z)
• OmniGAIA: Towards Native Omni-Modal AI Agents [103.79729735478924]
  We introduce a benchmark designed to evaluate omni-modal agents on tasks requiring deep reasoning and multi-turn tool execution.<n>We propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception.
  arXiv  Detail & Related papers  (2026-02-26T11:35:04Z)
• OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding [23.176694412214157]
  We introduce OmniAgent, a fully audio-guided active perception agent.<n>This paper demonstrates a paradigm shift from passive response generation to active multimodal inquiry.
  arXiv  Detail & Related papers  (2025-12-29T17:59:05Z)
• A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos [76.98722001848493]
  LongShOTBench is a diagnostic benchmark for long-form multimodal video understanding.<n>It includes open-ended, intent-driven questions; single- and multi-turn dialogues; and tasks requiring multimodal reasoning and agentic tool use.<n>LongShOTAgent is an agentic system that analyzes long videos via preprocessing, search, and iterative refinement.
  arXiv  Detail & Related papers  (2025-12-18T18:59:27Z)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs [72.425061028374]
  We introduce OmniVideoBench, a benchmark dedicated to assessing synergistic audio-visual understanding.<n> OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces.<n>We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-12T16:34:00Z)
• Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval [58.640807985155554]
  Video Moment Retrieval (VMR) aims to retrieve a specific moment semantically related to a given query.<n>Most existing VMR methods solely focus on the visual and textual modalities while neglecting the complementary but important audio modality.<n>We propose a novel Importance-aware Multi-Granularity fusion model (IMG), which learns to dynamically and selectively aggregate the audio-vision-text contexts for VMR.
  arXiv  Detail & Related papers  (2025-08-06T09:58:43Z)
• AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding [73.60257070465377]
  AdaVideoRAG is a novel framework that adapts retrieval based on query complexity using a lightweight intent classifier.<n>Our framework employs an Omni-Knowledge Indexing module to build hierarchical databases from text (captions, ASR, OCR), visual features, and semantic graphs.<n> Experiments demonstrate improved efficiency and accuracy for long-video understanding, with seamless integration into existing MLLMs.
  arXiv  Detail & Related papers  (2025-06-16T15:18:15Z)
• MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks [67.31276358668424]
  We introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer.<n> AVHaystacks is an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task.<n>We propose a model-agnostic, multi-agent framework to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystack
  arXiv  Detail & Related papers  (2025-06-08T06:34:29Z)
• OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup [50.70494796172493]
  We introduce Omni-modal Sound Separation (OmniSep), a novel framework capable of isolating clean soundtracks based on omni-modal queries. We introduce the Query-Mixup strategy, which blends query features from different modalities during training. We further enhance this flexibility by allowing queries to influence sound separation positively or negatively, facilitating the retention or removal of specific sounds.
  arXiv  Detail & Related papers  (2024-10-28T17:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.