A Simple Baseline for Streaming Video Understanding

• URL: http://arxiv.org/abs/2604.02317v1
• Date: Thu, 02 Apr 2026 17:58:04 GMT
• Title: A Simple Baseline for Streaming Video Understanding
• Abstract summary: We formalize a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM.<n>We evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench.
• Score: 44.77185258429313
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent streaming video understanding methods increasingly rely on complex memory mechanisms to handle long video streams. We challenge this trend with a simple finding: a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM already matches or surpasses published streaming models. We formalize this baseline as SimpleStream and evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench. Despite its simplicity, SimpleStream delivers consistently strong performance. With only 4 recent frames, it reaches 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench. Controlled ablations further show that the value of longer context is backbone-dependent rather than uniformly increasing with model scale, and reveal a consistent perception-memory trade-off: adding more historical context can improve recall, but often weakens real-time perception. This suggests that stronger memory, retrieval, or compression modules should not be taken as evidence of progress unless they clearly outperform SimpleStream under the same protocol. We therefore argue that future streaming benchmarks should separate recent-scene perception from long-range memory, so that performance improvements from added complexity can be evaluated more clearly.

Related papers

• StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs [53.38747124102543]
  StreamTTT writes long-range history into online-updated fast weights outside the attention context.<n>We train StreamTTT jointly on offline long-video QA and a newly constructed real-time QA corpus.<n>It outperforms the same-scale SimpleStream-4B by 1.4 points in real-time perception and 3.7 points in backward tracing.
  arXiv  Detail & Related papers  (2026-08-13T16:10:33Z)
• StreamFlow: Dynamic Memory Flows for Streaming Video Understanding [65.10324190760602]
  StreamFlow is an efficient visual memory framework that enables dynamic, on-demand access to historical visual information.<n>It achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench.<n>Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively.
  arXiv  Detail & Related papers  (2026-08-11T14:19:03Z)
• ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding [50.17161313335395]
  ObjectStream is a training-free framework that treats latent objects as memory anchors for streaming video understanding.<n>It preserves three complementary forms of evidence: persistent object histories, transient object changes, and recent visual context.<n>Experiments on online streaming and offline long-video benchmarks demonstrate both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-07-30T14:47:00Z)
• What Should a Streaming Video Model Remember? [28.189301152079214]
  Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets.<n>We propose textbfSelectStream, a selective latent-memory framework that keeps the current observation directly visible to a frozen VLM while exposing historical information only through a compact, query-conditioned evidence budget.<n> Experimental results show that SelectStream achieves strong online streaming performance and preserves general video understanding, reaching 82.67% on StreamingBench, 67.03% on OVO-Bench, and 74.4% average accuracy on offline video benchmarks
  arXiv  Detail & Related papers  (2026-06-15T07:50:19Z)
• Thinking in Streaming Video [30.61790766076081]
  ThinkStream is a framework for streaming video reasoning based on a Watch--Think--Speak paradigm.<n> Reasoning-Compressed Streaming Memory (RCSM) treats intermediate reasoning traces as compact semantic memory.<n>Experiments on multiple streaming video benchmarks show that ThinkStream significantly outperforms existing online video models.
  arXiv  Detail & Related papers  (2026-03-13T12:33:36Z)
• Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously [69.0264594684213]
  Video Streaming Thinking (VST) is a novel paradigm for streaming video understanding.<n>It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming.<n>VST improves timely comprehension and coherent cognition while preserving real-time responsiveness.
  arXiv  Detail & Related papers  (2026-03-12T17:59:51Z)
• Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory [50.30283773196725]
  Existing approaches rely on key-value caching to accumulate frame-level details over time, but use a limited number of tokens per frame.<n>We propose scaling the token budget to enable more granular-temporal understanding and reasoning.
  arXiv  Detail & Related papers  (2026-02-20T18:59:50Z)
• Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams [11.495597616926274]
  Event-VStream represents continuous video as a sequence of discrete, semantically coherent events.<n>System detects meaningful state transitions by integrating motion, semantic, and predictive cues.<n>System maintains around 70% GPT-5 win rate on 2-hour Ego4D streams.
  arXiv  Detail & Related papers  (2026-01-22T05:05:53Z)
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory [37.73273040737155]
  StreamForest is designed for streaming video understanding.<n>Fine-grained Spatiotemporal Window captures detailed short-term visual cues to improve current scene perception.<n>OnlineIT significantly boosts MLLM performance in both real-time perception and future prediction.
  arXiv  Detail & Related papers  (2025-09-29T14:53:57Z)
• OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding? [51.45196331624591]
  OVO-Bench is a novel benchmark for advanced online video understanding capability.<n>It consists of 12 tasks, featuring 644 unique videos and approximately human-curated 2,800 fine-grained meta-annotations with precise timestamps.<n> Evaluations of nine Video-LLMs reveal that, despite advancements on traditional benchmarks, current models struggle with online video understanding.
  arXiv  Detail & Related papers  (2025-01-09T19:00:01Z)
• Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams [78.72965584414368]
  We present Flash-VStream, a video-language model that simulates the memory mechanism of human. Compared to existing models, Flash-VStream achieves significant reductions in latency inference and VRAM consumption. We propose VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding.
  arXiv  Detail & Related papers  (2024-06-12T11:07:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.