Native Active Perception as Reasoning for Omni-Modal Understanding

• URL: http://arxiv.org/abs/2606.19341v1
• Date: Wed, 17 Jun 2026 17:59:56 GMT
• Title: Native Active Perception as Reasoning for Omni-Modal Understanding
• Abstract summary: We propose OmniAgent to formulate video understanding as a POMDP-based iterative Observation-Thought-Action cycle.<n> OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory.<n>We show that OmniAgent achieves state-of-the-art performance among open-source models.
• Score: 96.19360756363501
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5% vs. 47.3%).

Related papers

• VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding [61.00809248898572]
  Long video understanding is a problem in which an agent reads the video turn by turn along the temporal axis, deciding at each turn how fast to watch, what evidence to retain, when to revisit uncertain segments, and when to stop and answer.<n>We propose textbfVideoScout, a multi-turn reasoning agent that instantiates the SEA paradigm through adaptive reasoning pacing.<n>Experiments on long video understanding and reasoning benchmarks demonstrate that our 7B model achieves strong performance compared with existing trained 7B agentic models.
  arXiv  Detail & Related papers  (2026-09-14T14:04:03Z)
• Learning to Watch: Active Video Anomaly Understanding via Interleaved Policy Optimization [56.150184676745205]
  Video anomaly understanding (VAU) relies on sparse, context-dependent cues.<n>We propose $Anomtext-$, a closed-loop framework that reconceptualizes video understanding as an active sequential decision-making process.<n>Inspired by human video-reviewing behavior, this framework unifies internal cognitive reasoning and strategic evidence acquisition into an interleaved policy.
  arXiv  Detail & Related papers  (2026-07-01T08:41:49Z)
• MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction [76.4461698685681]
  Mini-o 4.5 is our latest effort towards human-level realtime streaming interaction.<n>It is based on Omni-CPM, a unified streaming framework that aligns omni-modal perception and outputs along a shared temporal axis.<n>With a total of 9B parameters, Mini-o 4.5 approaches Gemini 2.5 Flash in vision computation capabilities, delivering stateofthe-art open computation performance scale.
  arXiv  Detail & Related papers  (2026-04-30T04:05:43Z)
• A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Model [112.9420001646428]
  Vision-Language-Action (VLA) models have emerged as a powerful paradigm for open-world robot manipulation, but their practical deployment is often constrained by cost.<n>We present A1, a fully open-source and transparent VLA framework designed for low-cost, high- throughput inference.<n>A1 achieves state-of-the-art success rates while significantly reducing inference cost.
  arXiv  Detail & Related papers  (2026-04-07T10:18:40Z)
• EVA: Efficient Reinforcement Learning for End-to-End Video Agent [28.603844837930225]
  We present EVA, an efficient Reinforcement Learning framework for End-to-End Video Agent.<n> EVA autonomously decides what to watch, when to watch, and how to watch, achieving query-driven and efficient video understanding.
  arXiv  Detail & Related papers  (2026-03-24T08:06:29Z)
• Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation [95.89924101984566]
  We introduce OptimusVLA, a dual-memory VLA framework with Global Prior Memory (GPM) and Local Consistency Memory (LCM)<n>GPM replaces Gaussian noise with task-level priors retrieved from semantically similar trajectories.<n>LCM injects a learned consistency constraint that enforces temporal coherence and smoothness of trajectory.
  arXiv  Detail & Related papers  (2026-02-22T15:39:34Z)
• TIR-Flow: Active Video Search and Reasoning with Frozen VLMs [13.783722086940074]
  Large Video-Language Models (Video-LLMs) have achieved remarkable progress in perception, but their reasoning capabilities remain a bottleneck.<n>We propose TIR-Flow, a novel framework that shifts the paradigm from passive processing to active video searching and reasoning without additional data or parameter updating.
  arXiv  Detail & Related papers  (2026-01-07T13:54:44Z)
• OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding [23.176694412214157]
  We introduce OmniAgent, a fully audio-guided active perception agent.<n>This paper demonstrates a paradigm shift from passive response generation to active multimodal inquiry.
  arXiv  Detail & Related papers  (2025-12-29T17:59:05Z)
• VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception [50.446538409259524]
  Visual Test-Time Scaling (VTTS) is a novel approach to enhance MLLMs' reasoning via iterative inference during inference.<n>VTTS mimics humans' attention by focusing on high-confidence hierarchical-temporal regions, guided by updated textual predictions.<n>Our newly introduced Videochat-R1.5 model has achieved remarkable improvements, with an average increase of over 5%.
  arXiv  Detail & Related papers  (2025-09-25T12:46:46Z)
• ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding [71.654781631463]
  ReAgent-V is a novel agentic video understanding framework.<n>It integrates efficient frame selection with real-time reward generation during inference.<n>Extensive experiments on 12 datasets demonstrate significant gains in generalization and reasoning.
  arXiv  Detail & Related papers  (2025-06-02T04:23:21Z)
• VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoT [31.413204839972984]
  We propose a specialized chain-of-thought (CoT) process tailored for long video analysis.<n>Our uncertainty-aware CoT effectively mitigates noise from external tools, leading to more reliable outputs.<n>We implement our approach in a system called VideoAgent2, which also includes additional modules such as general context acquisition and specialized tool design.
  arXiv  Detail & Related papers  (2025-04-06T13:03:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.