WAT: Online Video Understanding Needs Watching Before Thinking

• URL: http://arxiv.org/abs/2603.13412v1
• Date: Thu, 12 Mar 2026 13:42:32 GMT
• Title: WAT: Online Video Understanding Needs Watching Before Thinking
• Abstract summary: WAT (Watching Before Thinking) is a two-stage framework for online video reasoning.<n>It separates processing into a query-independent watching stage and a query-triggered thinking stage.<n>It achieves state-of-the-art performance on online video benchmarks, including 77.7% accuracy on StreamingBench and 55.2% on OVO-Bench.
• Score: 25.136741695647213
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) have shown strong capabilities in image understanding, motivating recent efforts to extend them to video reasoning. However, existing Video LLMs struggle in online streaming scenarios, where long temporal context must be preserved under strict memory constraints. We propose WAT (Watching Before Thinking), a two-stage framework for online video reasoning. WAT separates processing into a query-independent watching stage and a query-triggered thinking stage. The watching stage builds a hierarchical memory system with a Short-Term Memory (STM) that buffers recent frames and a fixed-capacity Long-Term Memory (LTM) that maintains a diverse summary of historical content using a redundancy-aware eviction policy. In the thinking stage, a context-aware retrieval mechanism combines the query with the current STM context to retrieve relevant historical frames from the LTM for cross-temporal reasoning. To support training for online video tasks, we introduce WAT-85K, a dataset containing streaming-style annotations emphasizing real-time perception, backward tracing, and forecasting. Experiments show that WAT achieves state-of-the-art performance on online video benchmarks, including 77.7% accuracy on StreamingBench and 55.2% on OVO-Bench, outperforming existing open-source online Video LLMs while operating at real-time frame rates.

Related papers

• StreamFlow: Dynamic Memory Flows for Streaming Video Understanding [65.10324190760602]
  StreamFlow is an efficient visual memory framework that enables dynamic, on-demand access to historical visual information.<n>It achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench.<n>Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively.
  arXiv  Detail & Related papers  (2026-08-11T14:19:03Z)
• Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding [47.48582887762574]
  ReMem is a temporal granularity-adaptive selection framework for training-free LongVideoQA.<n>By explicitly preserving temporal information with memory mechanisms, ReMem suppresses redundancy and empowers MLLMs to perform robust multi-granular video reasoning.
  arXiv  Detail & Related papers  (2026-06-30T10:55:56Z)
• STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models [81.32710031596591]
  Video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames.<n>Existing methods built on vision-language models (LVLMs) often address this challenge by externalizing reasoning through chain-of-thought (CoT)<n>We propose STORMS, a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT.
  arXiv  Detail & Related papers  (2026-05-25T16:33:00Z)
• Thinking in Streaming Video [30.61790766076081]
  ThinkStream is a framework for streaming video reasoning based on a Watch--Think--Speak paradigm.<n> Reasoning-Compressed Streaming Memory (RCSM) treats intermediate reasoning traces as compact semantic memory.<n>Experiments on multiple streaming video benchmarks show that ThinkStream significantly outperforms existing online video models.
  arXiv  Detail & Related papers  (2026-03-13T12:33:36Z)
• video-SALMONN S: Streaming Audio-Visual LLMs Beyond Length Limits via Memory [51.03819128505358]
  Video-SALMONN S is first to process 3-hour videos at 1 FPS and 360p resolution under a fixed memory budget.<n>A test-time-training memory module continually updates token representations to capture long-range dependencies.<n>A prompt-dependent memory reader retrieves context-relevant content from fixed-size memory.
  arXiv  Detail & Related papers  (2025-10-13T08:20:15Z)
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory [37.73273040737155]
  StreamForest is designed for streaming video understanding.<n>Fine-grained Spatiotemporal Window captures detailed short-term visual cues to improve current scene perception.<n>OnlineIT significantly boosts MLLM performance in both real-time perception and future prediction.
  arXiv  Detail & Related papers  (2025-09-29T14:53:57Z)
• Episodic Memory Representation for Long-form Video Understanding [52.33907540905242]
  Large Video Language Models excel at general video understanding but struggle with long-form context window limits.<n>We introduce Video-EM, a training free framework inspired by the principles of human memory.<n>Video-EM achieves performance gains of 4-9 percent over respective baselines while utilizing fewer frames.
  arXiv  Detail & Related papers  (2025-08-13T04:33:07Z)
• HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding [14.464718780172582]
  We introduce HierarQ, a task-aware hierarchical Q-Former based framework that sequentially processes frames to bypass the need for frame sampling.<n>We introduce a lightweight two-stream language-guided feature modulator to incorporate task awareness in video understanding.<n>Extensive evaluations on 10 video benchmarks across video understanding, question answering, and captioning tasks demonstrate HierarQ's state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-11T16:21:23Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding [56.78088668917983]
  We introduce SVBench, a pioneering benchmark with temporal multi-turn question-answering chains.<n>We design a semi-automated annotation pipeline to obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos.<n>Our experimental results, obtained from 14 models in dialogue and streaming evaluations, reveal that while the closed-source GPT-4o outperforms others, most open-source LVLMs struggle with long-context streaming video understanding.
  arXiv  Detail & Related papers  (2025-02-15T14:29:44Z)
• Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams [78.72965584414368]
  We present Flash-VStream, a video-language model that simulates the memory mechanism of human. Compared to existing models, Flash-VStream achieves significant reductions in latency inference and VRAM consumption. We propose VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding.
  arXiv  Detail & Related papers  (2024-06-12T11:07:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.