Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously

• URL: http://arxiv.org/abs/2603.12262v1
• Date: Thu, 12 Mar 2026 17:59:51 GMT
• Title: Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously
• Abstract summary: Video Streaming Thinking (VST) is a novel paradigm for streaming video understanding.<n>It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming.<n>VST improves timely comprehension and coherent cognition while preserving real-time responsiveness.
• Score: 69.0264594684213
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online Video Large Language Models (VideoLLMs) play a critical role in supporting responsive, real-time interaction. Existing methods focus on streaming perception, lacking a synchronized logical reasoning stream. However, directly applying test-time scaling methods incurs unacceptable response latency. To address this trade-off, we propose Video Streaming Thinking (VST), a novel paradigm for streaming video understanding. It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming. This design improves timely comprehension and coherent cognition while preserving real-time responsiveness by amortizing LLM reasoning latency over video playback. Furthermore, we introduce a comprehensive post-training pipeline that integrates VST-SFT, which structurally adapts the offline VideoLLM to causal streaming reasoning, and VST-RL, which provides end-to-end improvement through self-exploration in a multi-turn video interaction environment. Additionally, we devise an automated training-data synthesis pipeline that uses video knowledge graphs to generate high-quality streaming QA pairs, with an entity-relation grounded streaming Chain-of-Thought to enforce multi-evidence reasoning and sustained attention to the video stream. Extensive evaluations show that VST-7B performs strongly on online benchmarks, e.g. 79.5% on StreamingBench and 59.3% on OVO-Bench. Meanwhile, VST remains competitive on offline long-form or reasoning benchmarks. Compared with Video-R1, VST responds 15.7 times faster and achieves +5.4% improvement on VideoHolmes, demonstrating higher efficiency and strong generalization across diverse video understanding tasks. Code, data, and models will be released at https://github.com/1ranGuan/VST.

Related papers

• Harnessing Streaming Video in the Wild [53.23721420272668]
  Vision-Language Models (VLMs) are increasingly required to process video streams in applications such as video-call assistants, live commentary, and embodied robots.<n>An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing.<n>ExistingVLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment.
  arXiv  Detail & Related papers  (2026-06-07T13:00:19Z)
• Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding [69.296913137409]
  We introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS)<n>We present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations.<n>First, the Frame-Driven Transition Controller (FDTC), makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent.<n>Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content.
  arXiv  Detail & Related papers  (2026-06-05T07:29:20Z)
• Thinking in Streaming Video [30.61790766076081]
  ThinkStream is a framework for streaming video reasoning based on a Watch--Think--Speak paradigm.<n> Reasoning-Compressed Streaming Memory (RCSM) treats intermediate reasoning traces as compact semantic memory.<n>Experiments on multiple streaming video benchmarks show that ThinkStream significantly outperforms existing online video models.
  arXiv  Detail & Related papers  (2026-03-13T12:33:36Z)
• Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models [25.48403877647779]
  Think While Watching is a memory-anchored streaming video reasoning framework.<n>It preserves continuous segment-level memory during multi-turn interaction.<n>It improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench.
  arXiv  Detail & Related papers  (2026-03-12T13:13:50Z)
• Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models [14.21980212001207]
  Motivated by the streaming nature of video data, we investigate two streaming reasoning paradigms for LVLMs.<n>To better match streaming inputs, we propose textbfThink-as-You-See (TaYS), a unified framework enabling true concurrent reasoning.
  arXiv  Detail & Related papers  (2026-03-03T11:24:55Z)
• StreamingVLM: Real-Time Understanding for Infinite Video Streams [23.94087606884915]
  StreamingVLM is a model designed for real-time, stable understanding of infinite visual input.<n>Our approach is a unified framework that aligns training with streaming inference.<n>On Inf-Streams-Eval, StreamingVLM achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100.
  arXiv  Detail & Related papers  (2025-10-10T17:59:58Z)
• SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding [56.78088668917983]
  We introduce SVBench, a pioneering benchmark with temporal multi-turn question-answering chains.<n>We design a semi-automated annotation pipeline to obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos.<n>Our experimental results, obtained from 14 models in dialogue and streaming evaluations, reveal that while the closed-source GPT-4o outperforms others, most open-source LVLMs struggle with long-context streaming video understanding.
  arXiv  Detail & Related papers  (2025-02-15T14:29:44Z)
• OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding? [51.45196331624591]
  OVO-Bench is a novel benchmark for advanced online video understanding capability.<n>It consists of 12 tasks, featuring 644 unique videos and approximately human-curated 2,800 fine-grained meta-annotations with precise timestamps.<n> Evaluations of nine Video-LLMs reveal that, despite advancements on traditional benchmarks, current models struggle with online video understanding.
  arXiv  Detail & Related papers  (2025-01-09T19:00:01Z)
• Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams [78.72965584414368]
  We present Flash-VStream, a video-language model that simulates the memory mechanism of human. Compared to existing models, Flash-VStream achieves significant reductions in latency inference and VRAM consumption. We propose VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding.
  arXiv  Detail & Related papers  (2024-06-12T11:07:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.