Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding

• URL: http://arxiv.org/abs/2606.25658v1
• Date: Wed, 24 Jun 2026 10:11:08 GMT
• Title: Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
• Abstract summary: CausalMem is dedicated to constructing a dynamic visual memory update mechanism.<n>It estimates the redundancy of visual tokens and updates the memory bank via an online semantic basis.<n>It achieves more than textbf20$times$ visual token compression ratio and only occupies about textbf82 MB storage.
• Score: 56.17043150838547
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Currently, streaming video understanding is still a daunting task for existing \emph{multimodal large language models} (MLLMs). Its difficulties not only lie in handling the ever-increasing video frames, but also in the unpredictability of future video content and input instructions. In this paper, we study this task from the perspective of constructing a dynamic but fixed-budget memory bank, and propose a novel and training-free approach termed \emph{\textbf{CausalMem}}. CausalMem is dedicated to constructing a dynamic visual memory update mechanism, thereby maximizing the amount of information in streaming video within a limited memory space, much like the human brain. In practice, CausalMem estimates the redundancy of visual tokens and updates the memory bank via an online semantic basis, which models the principal semantics of the observed video stream. To validate CausalMem, we apply it to two representative MLLMs, namely LLaVA-OneVision and Qwen2.5-VL respectively, and conduct extensive experiments on both streaming and offline video understanding benchmarks. The experimental results not only show the great advantages than existing methods under both streaming and offline settings, \emph{e.g.}, $+3.2\%$ and $+3.0\%$ average accuracy gains respectively, but also witness the superior semantic preservation for streaming videos, \emph{e.g.}, using 12$k$ token budgets to memorize hour-long streaming videos, which achieves more than \textbf{20$\times$} visual token compression ratio and only occupies about \textbf{82 MB} storage. \textbf{Our code} is given in \href{https://github.com/hktk07/CausalMem}{CausalMem}.

Related papers

• Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions [75.23170605943457]
  textbfmodel is a framework that decouples reasoning control from memory integration.<n>emphActive Thinking Decision Maker (ATDM) is a transparent reasoning controller that externalizes its decision process.<n>emphHierarchical Progressive Semantic Integration (HPSI) module acts as an efficient memory system.
  arXiv  Detail & Related papers  (2026-04-20T16:15:33Z)
• Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism [82.67996027633986]
  Long video understanding is a key challenge that plagues the advancement of emphMultimodal Large language Models (MLLMs)<n>In this paper, we study this problem from the perspective of visual memory mechanism, and proposed a novel and training-free approach, termed emphFlexible Memory (textbfFlexMem)<n>In principle, FlexMem aims to mimic human behavior of video watching, emphi.e., continually watching video content and recalling the most relevant memory fragments to answer the question.
  arXiv  Detail & Related papers  (2026-03-31T04:23:33Z)
• Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory [50.30283773196725]
  Existing approaches rely on key-value caching to accumulate frame-level details over time, but use a limited number of tokens per frame.<n>We propose scaling the token budget to enable more granular-temporal understanding and reasoning.
  arXiv  Detail & Related papers  (2026-02-20T18:59:50Z)
• HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding [92.59317281526239]
  HERMES is a training-free architecture for real-time and accurate understanding of video streams.<n>Hermes reuses a compact KV cache, enabling efficient streaming understanding under resource constraints.<n>Hermes achieves superior or comparable accuracy across all benchmarks, with up to 11.4% gains on streaming datasets.
  arXiv  Detail & Related papers  (2026-01-21T07:26:15Z)
• video-SALMONN S: Streaming Audio-Visual LLMs Beyond Length Limits via Memory [51.03819128505358]
  Video-SALMONN S is first to process 3-hour videos at 1 FPS and 360p resolution under a fixed memory budget.<n>A test-time-training memory module continually updates token representations to capture long-range dependencies.<n>A prompt-dependent memory reader retrieves context-relevant content from fixed-size memory.
  arXiv  Detail & Related papers  (2025-10-13T08:20:15Z)
• FastVID: Dynamic Density Pruning for Fast Video Large Language Models [38.267065642416554]
  We propose Density Pruning for Fast Video LLMs termed FastVID.<n>FastVID partitions videos into temporally ordered segments to preserve temporal structure.<n>Our method significantly reduces computational overhead while maintaining temporal and visual integrity.
  arXiv  Detail & Related papers  (2025-03-14T08:33:08Z)
• VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers [23.541896057977745]
  VideoScan is an efficient vision-language model (VLM) inference framework for real-time video interaction.<n>VideoScan employs a single semantic carrier token to represent each frame.
  arXiv  Detail & Related papers  (2025-03-12T13:30:40Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.