FOLIO: Focused Semantic Memory for Streaming Video Understanding

• URL: http://arxiv.org/abs/2607.13298v1
• Date: Tue, 14 Jul 2026 22:09:48 GMT
• Title: FOLIO: Focused Semantic Memory for Streaming Video Understanding
• Abstract summary: In online streaming video understanding, a video stream continues to arrive and queries may be issued at any time.<n>As this history grows with the stream, memory cost increases and many redundant visual details are retained.<n>We introduce FOLIO, a training-free focused semantic memory system that records important parts of the stream in higher detail.
• Score: 7.067883491096352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In online streaming video understanding, a video stream continues to arrive and queries may be issued at any time. Because streaming frames grow without bound, the system must continuously compress and retain information from the observed video prefix while future frames and future queries remain unknown. The core challenge is deciding what information to retain and how to organize the maintained history: as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes. To address this challenge, we introduce FOLIO, a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact. As the stream arrives, FOLIO updates memory at the segment level, guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache. At query time, lightweight hybrid retrieval combines direct matching over the structured memory with semantic query expansion. FOLIO achieves state-of-the-art performance, reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74.5 overall accuracy on StreamingBench, while substantially reducing the cost of maintaining streaming memory by reserving detailed records for focused entities and storing surrounding context compactly.

Related papers

• Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding [64.34948094316685]
  We introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize.<n>LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T17:28:14Z)
• Dynamic Hub-and-Spoke Memory for Streaming Video Understanding [36.38729564564716]
  Dynamic Hub-and-Spoke Memory (D-HSM) is a training-free framework that represents distant history as structured textual memory.<n>D-HSM turns selected historical video chunks into typed textual observations and stores them in an entity-centered hub-and-spoke memory.<n>When answering a question, D-HSM dynamically retrieves a compact question-aware memory subset, expands it through hub-and-spoke links, and combines it with the recent visual window for frozen-VLM answer prediction.
  arXiv  Detail & Related papers  (2026-08-31T06:02:28Z)
• RECAP-Forcing: Retaining Content Appearances for Long Video Generation [37.756464872090596]
  We propose RECAP-Forcing, organizing memory by appearance novelty.<n>A long video is not merely a sequence of frames, but an evolving cast of subjects, objects, and scenes.<n>As the video evolves, an optical-flow-based novelty bank extends the same principle by selectively retaining newly revealed content.
  arXiv  Detail & Related papers  (2026-08-27T06:24:59Z)
• StreamFlow: Dynamic Memory Flows for Streaming Video Understanding [65.10324190760602]
  StreamFlow is an efficient visual memory framework that enables dynamic, on-demand access to historical visual information.<n>It achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench.<n>Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively.
  arXiv  Detail & Related papers  (2026-08-11T14:19:03Z)
• ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding [50.17161313335395]
  ObjectStream is a training-free framework that treats latent objects as memory anchors for streaming video understanding.<n>It preserves three complementary forms of evidence: persistent object histories, transient object changes, and recent visual context.<n>Experiments on online streaming and offline long-video benchmarks demonstrate both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-07-30T14:47:00Z)
• What Should a Streaming Video Model Remember? [28.189301152079214]
  Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets.<n>We propose textbfSelectStream, a selective latent-memory framework that keeps the current observation directly visible to a frozen VLM while exposing historical information only through a compact, query-conditioned evidence budget.<n> Experimental results show that SelectStream achieves strong online streaming performance and preserves general video understanding, reaching 82.67% on StreamingBench, 67.03% on OVO-Bench, and 74.4% average accuracy on offline video benchmarks
  arXiv  Detail & Related papers  (2026-06-15T07:50:19Z)
• FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion [59.207505503284715]
  FadeMem organizes historical KV blocks into a temporal hierarchy under a fixed cache budget.<n>New history is inserted as fine-grained entries, while older adjacent entries are progressively merged.<n>Experiments show improved subject consistency, background stability, and temporal coherence over existing bounded-cache strategies.
  arXiv  Detail & Related papers  (2026-06-09T10:22:18Z)
• Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding [55.7992006853979]
  SAVEMem is a framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.<n>We present SAVEMem, a training-free dual-stage framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.
  arXiv  Detail & Related papers  (2026-05-08T15:40:40Z)
• WAT: Online Video Understanding Needs Watching Before Thinking [25.136741695647213]
  WAT (Watching Before Thinking) is a two-stage framework for online video reasoning.<n>It separates processing into a query-independent watching stage and a query-triggered thinking stage.<n>It achieves state-of-the-art performance on online video benchmarks, including 77.7% accuracy on StreamingBench and 55.2% on OVO-Bench.
  arXiv  Detail & Related papers  (2026-03-12T13:42:32Z)
• Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory [50.30283773196725]
  Existing approaches rely on key-value caching to accumulate frame-level details over time, but use a limited number of tokens per frame.<n>We propose scaling the token budget to enable more granular-temporal understanding and reasoning.
  arXiv  Detail & Related papers  (2026-02-20T18:59:50Z)
• WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning [66.24870234484668]
  We introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories.<n>WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks.
  arXiv  Detail & Related papers  (2025-12-02T05:14:52Z)
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory [37.73273040737155]
  StreamForest is designed for streaming video understanding.<n>Fine-grained Spatiotemporal Window captures detailed short-term visual cues to improve current scene perception.<n>OnlineIT significantly boosts MLLM performance in both real-time perception and future prediction.
  arXiv  Detail & Related papers  (2025-09-29T14:53:57Z)
• Mixture of Contexts for Long Video Generation [72.96361488755986]
  We recast long-context video generation as an internal information retrieval task.<n>We propose a simple, learnable sparse attention routing module, Mixture of Contexts (MoC), as an effective long-term memory retrieval engine.<n>As we scale the data and gradually sparsify the routing, the model allocates compute to salient history, preserving identities, actions, and scenes over minutes of content.
  arXiv  Detail & Related papers  (2025-08-28T17:57:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.