OVIBench: Benchmarking Online Video Question Answering under Interruption

• URL: http://arxiv.org/abs/2608.22279v1
• Date: Sun, 23 Aug 2026 08:12:08 GMT
• Title: OVIBench: Benchmarking Online Video Question Answering under Interruption
• Abstract summary: We introduce OVIBench, the first standardized benchmark for evaluating vision language models (VLMs)<n>OVIBench categorizes interruptions into three types: Cancellation, False Trigger, Correction and supports both open-ended and multiple-choice evaluations.<n>Experiments demonstrate that OVIBench effectively distinguishes models' interruption-handling abilities, especially in following correction requests.
• Score: 7.346987643902584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent vision language models (VLMs) have achieved strong progress in video understanding. However, most existing video QA research and benchmarks still follow an offline, single-round paradigm, overlooking realistic interactions where users may interrupt the model during answer generation. To address this gap, we formulate the task of Online Video Question Answering under Interruption and introduce OVIBench, the first standardized benchmark for evaluating VLMs in this setting. OVIBench categorizes interruptions into three types: Cancellation, False Trigger, Correction and supports both open-ended and multiple-choice evaluations. To enable large-scale and reproducible testing, we develop an offline simulation protocol that reproduces interruption during generation under a unified temporal setup, together with a multi-dimensional metric suite for assessing interruption understanding and response generation. Experiments demonstrate that OVIBench effectively distinguishes models' interruption-handling abilities, especially in following correction requests. Finally, we construct a train set OVI-Train for interruption-aware fine-tuning. Models fine-tuned on this dataset achieve significant gains on OVIBench, validating the effectiveness of our benchmark and data design. OVIBench, OVI-Train, and the evaluation code will be released.

Related papers

• CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models [41.60751475372144]
  Video Generation Models (VGMs) exhibit recurring failure modes on goal-directed tasks.<n>We propose VLM-VGM Collaborative Video Reasoning (CollabVR), a closed-loop framework that couples the VLM with the VGM at step-level critiques.<n>On Gen-ViRe and VBVR-Bench, CollabVR improves both open-source and closed-source VGMs over single-inference, Pass@$k$, and prior test-time scaling baselines at matched compute.
  arXiv  Detail & Related papers  (2026-05-09T06:39:17Z)
• VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models [39.78979236902648]
  Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames.<n>We propose VSAS-Bench, a new framework and benchmark for Visual Streaming Assistants.
  arXiv  Detail & Related papers  (2026-04-08T22:31:20Z)
• Robust Test-time Video-Text Retrieval: Benchmarking and Adapting for Query Shifts [28.52079785516312]
  Video-text retrieval (VTR) models are highly vulnerable to real-world query shifts.<n>We propose HAT-VTR (Hubness Alleviation for Test-time Video-Text Retrieval), as our baseline test-time adaptation framework.<n>Extensive experiments demonstrate that HAT-VTR substantially improves robustness, consistently outperforming methods across diverse query shift scenarios.
  arXiv  Detail & Related papers  (2026-02-15T05:57:44Z)
• VIPER: Process-aware Evaluation for Generative Video Reasoning [64.86465792516658]
  We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning.<n>Our experiments reveal that state-of-the-art video models achieve only about 20% POC@1.0 and exhibit a significant outcome-hacking.
  arXiv  Detail & Related papers  (2025-12-31T16:31:59Z)
• Stable Video Infinity: Infinite-Length Video Generation with Error Recycling [76.91310169118408]
  We propose Stable Video Infinity (SVI) that is able to generate infinite-length videos with high temporal consistency, plausible scene transitions, and controllable streaming storylines.<n> SVI incorporates Error-Recycling Fine-Tuning, a new type of efficient training that recycles the Diffusion Transformer's self-generated errors into supervisory prompts.<n>We evaluate SVI on three benchmarks, including consistent, creative, and conditional settings, thoroughly verifying its versatility and state-of-the-art role.
  arXiv  Detail & Related papers  (2025-10-10T09:45:46Z)
• Video-LLMs with Temporal Visual Screening [59.18455762289321]
  Temporal Visual Screening (TVS) is a new task that universally pre-processes video question answering and instruction tuning data.<n>TVS is formulated as a modular front-end adapter task that can be seamlessly integrated into both Video Instruction Tuning (training) and Video Question Answering (inference) pipelines.<n> Experiments demonstrate that incorporating TVS yields relative gains of 7.33% (training) and 34.6% (inference)
  arXiv  Detail & Related papers  (2025-08-27T14:33:32Z)
• AssistPDA: An Online Video Surveillance Assistant for Video Anomaly Prediction, Detection, and Analysis [52.261173507177396]
  We introduce AssistPDA, the first online video anomaly surveillance assistant (VAPDA) that unifies anomaly prediction, detection, and analysis (VAPDA) within a single framework.<n> AssistPDA enables real-time inference on streaming videos while supporting interactive user engagement.<n>We also introduce a novel event-level anomaly prediction task, enabling proactive anomaly forecasting before anomalies fully unfold.
  arXiv  Detail & Related papers  (2025-03-27T18:30:47Z)
• DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation [61.59996525424585]
  DIFFVSGG is an online VSGG solution that frames this task as an iterative scene graph update problem.<n>We unify the decoding of object classification, bounding box regression, and graph generation three tasks using one shared feature embedding.<n>DIFFVSGG further facilitates continuous temporal reasoning, where predictions for subsequent frames leverage results of past frames as the conditional inputs of LDMs.
  arXiv  Detail & Related papers  (2025-03-18T06:49:51Z)
• Live Video Captioning [0.6291443816903801]
  We introduce a groundbreaking paradigm: Live Video Captioning (LVC), where captions must be generated for video streams in an online manner.<n>We formally define the novel problem of LVC and propose innovative evaluation metrics specifically designed for this online scenario.<n>We present a new model that combines deformable transformers with temporal filtering, enabling effective captioning over video streams.
  arXiv  Detail & Related papers  (2024-06-20T11:25:16Z)
• Video Mask Transfiner for High-Quality Video Instance Segmentation [102.50936366583106]
  Video Mask Transfiner (VMT) is capable of leveraging fine-grained high-resolution features thanks to a highly efficient video transformer structure. Based on our VMT architecture, we design an automated annotation refinement approach by iterative training and self-correction. We compare VMT with the most recent state-of-the-art methods on the HQ-YTVIS, as well as the Youtube-VIS, OVIS and BDD100K MOTS.
  arXiv  Detail & Related papers  (2022-07-28T11:13:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.