CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models

• URL: http://arxiv.org/abs/2605.08735v1
• Date: Sat, 09 May 2026 06:39:17 GMT
• Title: CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models
• Abstract summary: Video Generation Models (VGMs) exhibit recurring failure modes on goal-directed tasks.<n>We propose VLM-VGM Collaborative Video Reasoning (CollabVR), a closed-loop framework that couples the VLM with the VGM at step-level critiques.<n>On Gen-ViRe and VBVR-Bench, CollabVR improves both open-source and closed-source VGMs over single-inference, Pass@$k$, and prior test-time scaling baselines at matched compute.
• Score: 41.60751475372144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent "Thinking with Video" approaches use Video Generation Models (VGMs) for visual reasoning by producing temporally coherent Chain-of-Frames as reasoning artifacts. Even strong VGMs, however, exhibit two recurring failure modes on goal-directed tasks: long-horizon drift on multi-step tasks and mid-clip simulation errors that compound. Both stem from the absence of explicit reasoning built upon the VGM's short-horizon visual prior, a role naturally filled by Vision-Language Models (VLMs), but where to place the VLM is non-trivial: upfront plans commit before any frame is generated and post-hoc critiques over whole videos intervene too late. We propose VLM-VGM Collaborative Video Reasoning (CollabVR), a closed-loop framework that couples the VLM with the VGM at step-level granularity: the VLM plans the immediate next action, inspects the clip the VGM generates, and folds the verifier's diagnosis directly into the next action prompt to repair detected failures. On Gen-ViRe and VBVR-Bench, CollabVR improves both open-source and closed-source VGMs over single-inference, Pass@$k$, and prior test-time scaling baselines at matched compute, with the largest gains on the hardest tasks. It also yields further improvements on top of a reasoning-fine-tuned VGM, indicating that step-level VLM supervision is orthogonal to and stackable with reasoning-oriented fine-tuning. We provide video samples and additional qualitative results at our project page: https://joow0n-kim.github.io/collabvr-project-page.

Related papers

• From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models [42.398664944109676]
  We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks.<n>We propose Vid-PRE (Video Reason Prompter and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM.
  arXiv  Detail & Related papers  (2026-09-10T08:40:58Z)
• VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning [133.66021945481876]
  We introduce VBVR-Pro, a testbed that makes native visual reasoning generation controllable.<n> VBVR-Pro turns visual reasoning into a controlled task space of 300 procedurally generated tasks.<n>We identify recurring failure modes of the prevalent VLM-as-a-judge paradigm.
  arXiv  Detail & Related papers  (2026-08-26T17:59:51Z)
• OVIBench: Benchmarking Online Video Question Answering under Interruption [7.346987643902584]
  We introduce OVIBench, the first standardized benchmark for evaluating vision language models (VLMs)<n>OVIBench categorizes interruptions into three types: Cancellation, False Trigger, Correction and supports both open-ended and multiple-choice evaluations.<n>Experiments demonstrate that OVIBench effectively distinguishes models' interruption-handling abilities, especially in following correction requests.
  arXiv  Detail & Related papers  (2026-08-23T08:12:08Z)
• VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization [34.39134324001484]
  We introduce a paradigm shift that transitions the role of Vision-Language Models (VLMs) to "teachers"<n>A VLM teacher extracts task-specific rules to differentiable rewards, formulate a V Reasoner via test-time online optimization of a LoRA module.<n>This strategy enables adaptive test-time optimization and extends the reasoning capabilities beyond the V's intrinsic boundaries.
  arXiv  Detail & Related papers  (2026-06-01T17:54:26Z)
• VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis [23.34626996348395]
  VGenSTBench is a video benchmark that employs generative models to actively synthesize, highly controlled and diverse evaluation scenarios.<n>We establish a comprehensive 3x2x2 video taxonomy, encompassing Spatial Scale Scene, Perspective Dynamics to span diverse scenarios.
  arXiv  Detail & Related papers  (2026-05-21T14:48:35Z)
• TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions [52.61846373082384]
  Shot Transition Detection (STD) inherently struggles with complex transitions.<n>We propose TransVLM, a Vision-Language Model (VLM) framework for STD.<n>Extensive experiments demonstrate that TransVLM achieves superior overall performance.
  arXiv  Detail & Related papers  (2026-04-30T15:05:06Z)
• GridVAD: Open-Set Video Anomaly Detection via Spatial Reasoning over Stratified Frame Grids [0.3624700141426058]
  Vision-Language Models (VLMs) are powerful open-set reasoners, yet their direct use as anomaly detectors in video surveillance is fragile.<n>We argue the problem is not the VLM itself but how it is used.<n>We instantiate this propose-ground-propagate principle in GridVAD, a training-free pipeline that produces pixel-level anomaly masks.
  arXiv  Detail & Related papers  (2026-03-26T14:08:41Z)
• Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders [40.81958598891815]
  Vision Language Model (VLM) development has largely relied on scaling model size.<n>We present Penguin-VL, whose vision encoder is from a text-only LLM.<n>Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining.
  arXiv  Detail & Related papers  (2026-03-06T18:58:04Z)
• VIPER: Process-aware Evaluation for Generative Video Reasoning [64.86465792516658]
  We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning.<n>Our experiments reveal that state-of-the-art video models achieve only about 20% POC@1.0 and exhibit a significant outcome-hacking.
  arXiv  Detail & Related papers  (2025-12-31T16:31:59Z)
• TRANSPORTER: Transferring Visual Semantics from VLM Manifolds [56.749972238005604]
  This paper introduces a logits-to-video (L2V) task alongside a model-independent approach, TRANSPORTER, to generate videos.<n> TRANSPORTER learns an optimal transport coupling to VLM's high-semantic embedding spaces.<n>In turn, logit scores define embedding directions for conditional video generation.
  arXiv  Detail & Related papers  (2025-11-23T09:12:48Z)
• Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding [56.45689495743107]
  Vgent is a graph-based retrieval-reasoning-augmented generation framework to enhance LVLMs for long video understanding.<n>We evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-15T19:14:58Z)
• Bridging Vision Language Models and Symbolic Grounding for Video Question Answering [4.215692222461999]
  Video Question Answering (VQA) requires models to reason over spatial, temporal, and causal cues in videos.<n>Recent vision language models (VLMs) achieve strong results but often rely on shallow correlations, leading to weak temporal grounding and limited interpretability.<n>We study symbolic scene graphs (SGs) as intermediate grounding signals for VQA.<n>We introduce SG-VLM, a modular framework that integrates frozen VLMs with scene graph grounding via prompting and visual localization.
  arXiv  Detail & Related papers  (2025-09-15T12:35:56Z)
• Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought [58.321044666612174]
  Vad-R1 is an end-to-end MLLM-based framework for Video Anomaly Reasoning.<n>We design a Perception-to-Cognition Chain-of-Thought (P2C-CoT) that simulates the human process of recognizing anomalies.<n>We also propose an improved reinforcement learning algorithm AVA-GRPO, which explicitly incentivizes the anomaly reasoning capability of MLLMs.
  arXiv  Detail & Related papers  (2025-05-26T12:05:16Z)
• VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models [29.706347050700867]
  We introduce a novel benchmark named Video-based long-form Causal Reasoning (VCRBench)<n>VCRBench tests whether Large Video Language Models (LVLMs) can identify, reason about, and correctly sequence the events needed to accomplish a specific goal.<n>We propose Recognition-Reasoning Decomposition (RRD), a modular approach that breaks video-based causal reasoning into two sub-tasks of video recognition and causal reasoning.
  arXiv  Detail & Related papers  (2025-05-13T11:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.