What Should a Streaming Video Model Remember?

• URL: http://arxiv.org/abs/2606.16353v1
• Date: Mon, 15 Jun 2026 07:50:19 GMT
• Title: What Should a Streaming Video Model Remember?
• Abstract summary: Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets.<n>We propose textbfSelectStream, a selective latent-memory framework that keeps the current observation directly visible to a frozen VLM while exposing historical information only through a compact, query-conditioned evidence budget.<n> Experimental results show that SelectStream achieves strong online streaming performance and preserves general video understanding, reaching 82.67% on StreamingBench, 67.03% on OVO-Bench, and 74.4% average accuracy on offline video benchmarks
• Score: 28.189301152079214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets. Existing methods address this by adding memory banks, retrieval modules, or visual token compression to preserve long-range history. However, strong recent-window baselines show that indiscriminate history injection can dilute current-scene perception, suggesting that the key challenge is not whether to use memory, but how to allocate it selectively. We formulate this as budgeted online latent evidence allocation and propose \textbf{SelectStream}, a selective latent-memory framework that keeps the current observation directly visible to a frozen VLM while exposing historical information only through a compact, query-conditioned evidence budget. Three coordinated mechanisms govern when to write, what to preserve, and how to retrieve: surprise-driven adaptive windowing, priority-preserving consolidation, and query-conditioned graph reasoning over a fixed-capacity latent memory graph. Retrieved evidence is calibrated and injected as latent tokens for answer generation, without replaying frames or growing the context with stream length. Experimental results show that SelectStream achieves strong online streaming performance and preserves general video understanding, reaching 82.67\% on StreamingBench, 67.03\% on OVO-Bench, and 74.4\% average accuracy on offline video benchmarks, while outperforming strong recent-window baselines and prior streaming memory methods.

Related papers

• Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding [64.34948094316685]
  We introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize.<n>LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T17:28:14Z)
• LayerRecall: A State-Conditioned Memory Router for Long-Horizon Consistency in Video Generation [22.878882105342182]
  Autoregressive video diffusion produces chunks from a bounded recent context.<n>Recency-based caching evicts historical cues needed when subjects, objects, scenes, or attributes reappear.<n>LayerRecall retrieves relevant historical K/V states and injects them only into backbone-specific memory-sensitive layers.
  arXiv  Detail & Related papers  (2026-08-28T15:49:13Z)
• StreamFlow: Dynamic Memory Flows for Streaming Video Understanding [65.10324190760602]
  StreamFlow is an efficient visual memory framework that enables dynamic, on-demand access to historical visual information.<n>It achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench.<n>Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively.
  arXiv  Detail & Related papers  (2026-08-11T14:19:03Z)
• Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding [56.17043150838547]
  CausalMem is dedicated to constructing a dynamic visual memory update mechanism.<n>It estimates the redundancy of visual tokens and updates the memory bank via an online semantic basis.<n>It achieves more than textbf20$times$ visual token compression ratio and only occupies about textbf82 MB storage.
  arXiv  Detail & Related papers  (2026-06-24T10:11:08Z)
• TetherCache: Stabilizing Autoregressive Long-Form Video Generation with Gated Recall and Trusted Alignment [51.33418612284208]
  We propose TetherCache, a training-free and plug-and-play cache management strategy for drift-resistant long video generation.<n>Gated Recall with Attention-Diversity Balancing selects long-range memory frames using a gated score.<n>TAME lightly edits newly recalled memory tokens by aligning their statistics to a trusted context distribution.
  arXiv  Detail & Related papers  (2026-06-11T08:16:08Z)
• Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding [55.7992006853979]
  SAVEMem is a framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.<n>We present SAVEMem, a training-free dual-stage framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.
  arXiv  Detail & Related papers  (2026-05-08T15:40:40Z)
• Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions [75.23170605943457]
  textbfmodel is a framework that decouples reasoning control from memory integration.<n>emphActive Thinking Decision Maker (ATDM) is a transparent reasoning controller that externalizes its decision process.<n>emphHierarchical Progressive Semantic Integration (HPSI) module acts as an efficient memory system.
  arXiv  Detail & Related papers  (2026-04-20T16:15:33Z)
• A Simple Baseline for Streaming Video Understanding [44.77185258429313]
  We formalize a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM.<n>We evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench.
  arXiv  Detail & Related papers  (2026-04-02T17:58:04Z)
• WAT: Online Video Understanding Needs Watching Before Thinking [25.136741695647213]
  WAT (Watching Before Thinking) is a two-stage framework for online video reasoning.<n>It separates processing into a query-independent watching stage and a query-triggered thinking stage.<n>It achieves state-of-the-art performance on online video benchmarks, including 77.7% accuracy on StreamingBench and 55.2% on OVO-Bench.
  arXiv  Detail & Related papers  (2026-03-12T13:42:32Z)
• Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory [50.30283773196725]
  Existing approaches rely on key-value caching to accumulate frame-level details over time, but use a limited number of tokens per frame.<n>We propose scaling the token budget to enable more granular-temporal understanding and reasoning.
  arXiv  Detail & Related papers  (2026-02-20T18:59:50Z)
• MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives [54.07515675393396]
  Existing solutions maintain the memory by compressing historical frames with predefined strategies.<n>We propose MemFlow to address this problem.<n>MemFlow achieves outstanding long-context consistency with negligible burden.
  arXiv  Detail & Related papers  (2025-12-16T18:59:59Z)
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory [37.73273040737155]
  StreamForest is designed for streaming video understanding.<n>Fine-grained Spatiotemporal Window captures detailed short-term visual cues to improve current scene perception.<n>OnlineIT significantly boosts MLLM performance in both real-time perception and future prediction.
  arXiv  Detail & Related papers  (2025-09-29T14:53:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.