StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

• URL: http://arxiv.org/abs/2603.21493v1
• Date: Mon, 23 Mar 2026 02:31:11 GMT
• Title: StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding
• Abstract summary: StreamingEval is a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints.<n>We adopt a fixed-capacity memory bank to normalize historical visual context, and jointly evaluate accessible visual encoding efficiency, text decoding latency, and task performance.
• Score: 26.728635060807377
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at https://github.com/wwgTang-111/StreamingEval1.

Related papers

• Harnessing Streaming Video in the Wild [53.23721420272668]
  Vision-Language Models (VLMs) are increasingly required to process video streams in applications such as video-call assistants, live commentary, and embodied robots.<n>An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing.<n>ExistingVLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment.
  arXiv  Detail & Related papers  (2026-06-07T13:00:19Z)
• Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects [26.144665202231014]
  Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance.<n>We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding, prefilling, and decoding.
  arXiv  Detail & Related papers  (2026-04-07T07:44:11Z)
• Exploring Real-Time Super-Resolution: Benchmarking and Fine-Tuning for Streaming Content [1.1926989512978123]
  We introduce a comprehensive dataset - StreamSR - sourced from YouTube.<n>We benchmark 11 state-of-the-art real-time super-resolution models to evaluate their performance for the streaming use-case.<n>We propose EfRLFN, an efficient real-time model that integrates Efficient Channel Attention and a hyperbolic tangent activation function.
  arXiv  Detail & Related papers  (2026-02-11T20:22:06Z)
• StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos [128.45606644157]
  StreamGaze is the first benchmark to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos.<n>We develop a gaze-video QA generation pipeline that aligns egocentric videos with raw gaze trajectories.<n>We observe substantial performance gaps between state-of-the-art MLLMs and human performance.
  arXiv  Detail & Related papers  (2025-12-01T14:15:44Z)
• StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding [52.55809460075286]
  We propose a StreamAgent that anticipates the temporal intervals and spatial regions expected to contain future task-relevant information.<n>We integrate question semantics and historical observations through prompting the anticipatory agent to anticipate the temporal progression of key events.<n>Our method outperforms existing methods in response accuracy and real-time efficiency, highlighting its practical value for real-world streaming scenarios.
  arXiv  Detail & Related papers  (2025-08-03T18:15:42Z)
• ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning [68.76048244253582]
  We introduce ViaRL, the first framework to leverage rule-based reinforcement learning (RL) for optimizing frame selection in video understanding.<n>ViaRL utilizes the answer accuracy of a downstream model as a reward signal to train a frame selector through trial-and-error.<n>ViaRL consistently delivers superior temporal grounding performance and robust generalization across diverse video understanding tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:29:40Z)
• StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition [20.608124640950276]
  We introduce StreamMind, a video LLM framework that achieves ultra-FPS streaming video processing (100 fps on a single A100)<n>We propose a novel perception-cognition intertemporal paradigm named ''event-gated LLM invocation''<n> Experiments on Ego4D and SoccerNet streaming tasks, as well as standard offline benchmarks, demonstrate state-of-the-art performance in both model capability and real-time efficiency.
  arXiv  Detail & Related papers  (2025-03-08T13:44:38Z)
• Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams [78.72965584414368]
  We present Flash-VStream, a video-language model that simulates the memory mechanism of human. Compared to existing models, Flash-VStream achieves significant reductions in latency inference and VRAM consumption. We propose VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding.
  arXiv  Detail & Related papers  (2024-06-12T11:07:55Z)
• A Coding Framework and Benchmark towards Low-Bitrate Video Understanding [63.05385140193666]
  We propose a traditional-neural mixed coding framework that takes advantage of both traditional codecs and neural networks (NNs) The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved. We build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach.
  arXiv  Detail & Related papers  (2022-02-06T16:29:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.