Don't Pause! Every prediction matters in a streaming video

• URL: http://arxiv.org/abs/2604.24317v1
• Date: Mon, 27 Apr 2026 11:07:03 GMT
• Title: Don't Pause! Every prediction matters in a streaming video
• Abstract summary: We introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception and assistive capabilities.<n> SPOT-Bench comes with Timeliness-F1, a metric that measures streaming predictions by their temporal precision and balanced coverage across the entire video.<n>Our benchmark reveals: (i) offline models detect events reliably but spam predictions unprompted; (ii) post-training for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency.
• Score: 55.509551643600794
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Streaming video models should respond the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming predictions untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception and assistive capabilities required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming predictions by their temporal precision and balanced coverage across the entire video. Our benchmark reveals: (i) offline models detect events reliably but spam predictions unprompted; (ii) post-training for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency. These findings motivate AsynKV, a training-free streaming adaptation of offline models, that retains their event perception while improving their streaming behavior. AsynKV features a long-short term memory, utilized efficiently by scaling compute during dead-time. It serves as a strong baseline on SPOT-Bench, outperforming existing streaming models, and achieves state-of-the-art on retrospective benchmarks.

Related papers

• StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs [53.38747124102543]
  StreamTTT writes long-range history into online-updated fast weights outside the attention context.<n>We train StreamTTT jointly on offline long-video QA and a newly constructed real-time QA corpus.<n>It outperforms the same-scale SimpleStream-4B by 1.4 points in real-time perception and 3.7 points in backward tracing.
  arXiv  Detail & Related papers  (2026-08-13T16:10:33Z)
• StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video [53.82672457255517]
  Proactive streaming video understanding requires models to continuously process video streams and decide when to respond, rather than merely what to respond.<n>Existing benchmarks largely follow a "see-then-answer" paradigm, where responses are triggered only after explicit evidence appears.<n>We introduce StreamPro, a new benchmark that evaluates streaming models from three complementary perspectives: Perception Understanding, Temporal Reasoning, and Proactive Agency.
  arXiv  Detail & Related papers  (2026-05-11T05:01:15Z)
• Towards Generative Predictive Display for Vision-Based Teleoperation: A Zero-Shot Benchmark of Off-the-Shelf Video Models [0.0]
  This paper presents a benchmark of off-the-shelf generative video models for short-horizon predictive display.<n>Performance is assessed using prediction accuracy, per-rollout latency, peak GPU memory usage, and temporal error evolution.<n>Findings highlight a gap between general-purpose generative video synthesis and the requirements of predictive display in teleoperation.
  arXiv  Detail & Related papers  (2026-05-10T17:36:22Z)
• Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously [69.0264594684213]
  Video Streaming Thinking (VST) is a novel paradigm for streaming video understanding.<n>It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming.<n>VST improves timely comprehension and coherent cognition while preserving real-time responsiveness.
  arXiv  Detail & Related papers  (2026-03-12T17:59:51Z)
• StreamReady: Learning What to Answer and When in Long Streaming Videos [28.99829321053373]
  We introduce a readiness-aware formulation of streaming video understanding with the Answer Readiness Score (ARS)<n>ARS defines an effective accuracy that measures not just whether a model is right, but whether it answers at the appropriate moment.<n>Building on this formulation, we introduce StreamReady, a framework to unify temporal reasoning with on-time answering.
  arXiv  Detail & Related papers  (2026-03-09T17:02:44Z)
• WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs [37.61875409530676]
  We present WeaveTime, a simple, efficient, and model agnostic framework that first teaches order and then uses order.<n>At inference, a Past-Current Dynamic Focus Cache performs uncertainty triggered, coarse-to-fine retrieval, expanding history only when needed.<n>These results establish WeaveTime as a practical path toward time aware stream Video-LLMs under strict online, time causal constraints.
  arXiv  Detail & Related papers  (2026-02-25T17:45:45Z)
• OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding? [51.45196331624591]
  OVO-Bench is a novel benchmark for advanced online video understanding capability.<n>It consists of 12 tasks, featuring 644 unique videos and approximately human-curated 2,800 fine-grained meta-annotations with precise timestamps.<n> Evaluations of nine Video-LLMs reveal that, despite advancements on traditional benchmarks, current models struggle with online video understanding.
  arXiv  Detail & Related papers  (2025-01-09T19:00:01Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• Real-time Object Detection for Streaming Perception [84.2559631820007]
  Streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. We build a simple and effective framework for streaming perception. Our method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline.
  arXiv  Detail & Related papers  (2022-03-23T11:33:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.