OmniReasoner: Thinking with Long Audio-Video via Native Tool Use

• URL: http://arxiv.org/abs/2607.19339v1
• Date: Tue, 21 Jul 2026 17:57:50 GMT
• Title: OmniReasoner: Thinking with Long Audio-Video via Native Tool Use
• Abstract summary: We introduce OmniReasoner, a tool-use framework for thinking with long audio-video.<n>It learns to decide whether and where to call a zoom-in tool before answering.<n>TimeAnchor keeps the tool's temporal argument valid and round-trip-consistent across granularities.
• Score: 25.032571988845262
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering. Because the model observes different sampling granularities before and after this call -- a sparse global preview and a denser local clip -- we introduce TimeAnchor, which keeps the tool's temporal argument valid and round-trip-consistent across these granularities, rather than tied to frame indices from a particular sampling rate. To make this tool-use behavior trainable without expensive manual interval annotation, we build a Temporal Augmented Data Engine that synthesizes tool-use post-training trajectories by video editing and composition. Experiments across omnimodal and video benchmarks show that OmniReasoner improves both answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. Code is available at https://github.com/RockyChen0205/OmniReasoner.

Related papers

• GigaChat Audio: Time-aware Large Audio Language Model [65.51589021112385]
  We present a time-aware audio LLM that answers questions with explicit timestamps over up to 120 minutes of input.<n>Our approach interleaves periodic time markers with continuous audio tokens using large-scale synthetic supervision from a cascaded pipeline.
  arXiv  Detail & Related papers  (2026-07-11T16:34:25Z)
• AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression [22.818321106501788]
  AVOC is a framework for long-form audio-video understanding in Omni-modal Large Language Models.<n>AVOC achieves state-of-the-art performance on long-form audio-video benchmarks.
  arXiv  Detail & Related papers  (2026-06-23T08:06:58Z)
• LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs [68.35684758116453]
  This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics.<n>We rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception.<n>Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%.
  arXiv  Detail & Related papers  (2026-03-19T17:58:13Z)
• OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering [12.963349283282447]
  We propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning.<n>It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks.<n>It uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries.
  arXiv  Detail & Related papers  (2026-02-03T16:28:24Z)
• A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos [76.98722001848493]
  LongShOTBench is a diagnostic benchmark for long-form multimodal video understanding.<n>It includes open-ended, intent-driven questions; single- and multi-turn dialogues; and tasks requiring multimodal reasoning and agentic tool use.<n>LongShOTAgent is an agentic system that analyzes long videos via preprocessing, search, and iterative refinement.
  arXiv  Detail & Related papers  (2025-12-18T18:59:27Z)
• LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling [87.98096428508181]
  LongVT is an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought.<n>We exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames.<n>Our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2025-11-25T19:22:48Z)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs [72.425061028374]
  We introduce OmniVideoBench, a benchmark dedicated to assessing synergistic audio-visual understanding.<n> OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces.<n>We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-12T16:34:00Z)
• Iterative Zoom-In: Temporal Interval Exploration for Long Video Understanding [18.027290155746112]
  Temporal Search is a training-free framework that enables MLLMs to explore temporal regions for improved long video understanding iteratively.<n>It is based on a key observation: the model's generation confidence across different temporal intervals is highly correlated with prediction accuracy.<n>It refines the focus of the model by iteratively shifting attention to more fine-grained temporal intervals, improving its understanding of long videos.
  arXiv  Detail & Related papers  (2025-06-28T15:24:05Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.