StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video

• URL: http://arxiv.org/abs/2605.16381v1
• Date: Mon, 11 May 2026 05:01:15 GMT
• Title: StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video
• Abstract summary: Proactive streaming video understanding requires models to continuously process video streams and decide when to respond, rather than merely what to respond.<n>Existing benchmarks largely follow a "see-then-answer" paradigm, where responses are triggered only after explicit evidence appears.<n>We introduce StreamPro, a new benchmark that evaluates streaming models from three complementary perspectives: Perception Understanding, Temporal Reasoning, and Proactive Agency.
• Score: 53.82672457255517
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Proactive streaming video understanding requires models to continuously process video streams and decide when to respond, rather than merely what to respond. This naturally introduces a decision-making problem under partial observations, where models must balance early prediction against sufficient evidence. However, existing benchmarks largely follow a "see-then-answer" paradigm, where responses are triggered only after explicit evidence appears, effectively reducing proactive reasoning to delayed perception. As a result, they fail to evaluate a model's ability to make timely and reliable decisions under incomplete observations. Moreover, training proactive models is inherently challenging due to the extreme imbalance between silence and response signals in streaming trajectories, as well as the need to jointly optimize response correctness and timing. To address these challenges, we introduce StreamPro-Bench, a new benchmark that evaluates streaming models from three complementary perspectives: Perception Understanding, Temporal Reasoning, and Proactive Agency, where the last measures a model's ability to make early yet reliable decisions under partial observations. We further propose StreamPro, a two-stage training framework for proactive learning. First, we introduce CB-Stream Loss to mitigate the severe supervision imbalance during supervised fine-tuning (SFT). Then, we apply Group Relative Policy Optimization (GRPO) with a multi-grained reward design that involves both turn-level and trajectory-level rewards. Experiments show that StreamPro significantly improves proactive performance. On StreamPro-Bench, it achieves 41.5, substantially outperforming the previous best (10.4), while also maintaining strong performance on real-time streaming benchmarks, achieving 78.9 on StreamingBench-RTVU.

Related papers

• AdaSR: Adaptive Streaming Reasoning with Hierarchical Relative Policy Optimization [12.363024790235158]
  AdaSR is an adaptive streaming reasoning framework that enables models to reason during input streaming and perform final deliberation once the stream is complete.<n>HRPO integrates format, accuracy, and adaptive thinking rewards to enforce valid reasoning protocols, preserve final task performance, and encourage latency-aware computation allocation.<n> Experiments show that AdaSR achieves a better balance among reasoning accuracy, computational efficiency, and streaming latency compared with supervised fine-tuning baseline.
  arXiv  Detail & Related papers  (2026-06-12T17:56:25Z)
• Don't Pause! Every prediction matters in a streaming video [55.509551643600794]
  We introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception and assistive capabilities.<n> SPOT-Bench comes with Timeliness-F1, a metric that measures streaming predictions by their temporal precision and balanced coverage across the entire video.<n>Our benchmark reveals: (i) offline models detect events reliably but spam predictions unprompted; (ii) post-training for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency.
  arXiv  Detail & Related papers  (2026-04-27T11:07:03Z)
• SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting [53.74101174559609]
  We propose a novel streaming-based motion forecasting framework that explicitly focuses on evolving scenes.<n>Our method incrementally processes incoming observation windows and leverages an instance-aware context streaming to maintain and update latent agent representations.<n>Our model achieves state-of-the-art performance in streaming inference on the Argoverse 2 multi-agent benchmark, while maintaining minimal latency, highlighting its suitability for real-world deployment.
  arXiv  Detail & Related papers  (2026-03-30T06:47:19Z)
• STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding [77.20037111885226]
  Real-world deployments increasingly require streaming perception and proactive interaction.<n>In this work, we revisit proactive activation in streaming video as a structured sequence modeling problem.<n>We propose STRIDE, which employs a lightweight masked diffusion module at the activation interface to jointly predict and progressively refine activation signals.
  arXiv  Detail & Related papers  (2026-03-29T09:23:45Z)
• Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding [19.76957611582713]
  Em-Garde is a novel framework that decouples semantic understanding from streaming perception.<n>Experiments on StreamingBench and OVO-Bench demonstrate consistent improvements over prior models in proactive response accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-03-19T15:47:50Z)
• StreamReady: Learning What to Answer and When in Long Streaming Videos [28.99829321053373]
  We introduce a readiness-aware formulation of streaming video understanding with the Answer Readiness Score (ARS)<n>ARS defines an effective accuracy that measures not just whether a model is right, but whether it answers at the appropriate moment.<n>Building on this formulation, we introduce StreamReady, a framework to unify temporal reasoning with on-time answering.
  arXiv  Detail & Related papers  (2026-03-09T17:02:44Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• StreamYOLO: Real-time Object Detection for Streaming Perception [84.2559631820007]
  We endow the models with the capacity of predicting the future, significantly improving the results for streaming perception. We consider multiple velocities driving scene and propose Velocity-awared streaming AP (VsAP) to jointly evaluate the accuracy. Our simple method achieves the state-of-the-art performance on Argoverse-HD dataset and improves the sAP and VsAP by 4.7% and 8.2% respectively.
  arXiv  Detail & Related papers  (2022-07-21T12:03:02Z)
• Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection [3.884530687475798]
  Streaming BERT-based sequence tagging model is capable of detecting disfluencies in real-time. Model attains state-of-the-art latency and stability scores when compared with recent work on incremental disfluency detection.
  arXiv  Detail & Related papers  (2022-05-02T02:13:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.