VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization

• URL: http://arxiv.org/abs/2606.02564v1
• Date: Mon, 01 Jun 2026 17:54:26 GMT
• Title: VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization
• Abstract summary: We introduce a paradigm shift that transitions the role of Vision-Language Models (VLMs) to "teachers"<n>A VLM teacher extracts task-specific rules to differentiable rewards, formulate a V Reasoner via test-time online optimization of a LoRA module.<n>This strategy enables adaptive test-time optimization and extends the reasoning capabilities beyond the V's intrinsic boundaries.
• Score: 34.39134324001484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The recent "Reasoning with Video" paradigm utilizes Video Generation Models (VGMs) to generate temporally coherent visual trajectories to complete reasoning tasks. Although state-of-the-art VGMs excel at visual quality, they often struggle to understand and follow task-specific rules, leading to logical failures across diverse reasoning scenarios. Existing efforts try to utilize Vision-Language Models (VLMs) as problem pre-solvers to produce or refine textual guidance for the VGM. However, textual descriptions fail to capture intricate spatiotemporal details, and VGMs often struggle to faithfully execute fine-grained or long-tail instructions even with a valid plan. While VLMs struggle as solvers, they possess strong perception capabilities to evaluate process-constraint satisfaction and final-goal achievement. Leveraging this strength, we introduce a paradigm shift that transitions the role of VLMs to "teachers". Specifically, a VLM teacher extracts task-specific rules to formulate differentiable rewards, guiding a VGM Reasoner via test-time online optimization of a lightweight LoRA module. This strategy enables adaptive test-time optimization and extends the reasoning capabilities beyond the VGM's intrinsic boundaries. Evaluations on symbolic (VBVR-Bench) and general-purpose (RULER-Bench) video reasoning benchmarks show that the proposed method yields a 16.7-point average performance gain, outperforming the VLM-as-Solver paradigm (+0.4 points) and Best-of-N scaling (+2.2 points) by a large margin at comparable test-time cost. These findings reveal that integrating VLMs as test-time teachers offers a promising paradigm for achieving generalizable video reasoning. Project Page: https://VLM-as-Teacher.github.io/

Related papers

• From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models [42.398664944109676]
  We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks.<n>We propose Vid-PRE (Video Reason Prompter and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM.
  arXiv  Detail & Related papers  (2026-09-10T08:40:58Z)
• CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models [41.60751475372144]
  Video Generation Models (VGMs) exhibit recurring failure modes on goal-directed tasks.<n>We propose VLM-VGM Collaborative Video Reasoning (CollabVR), a closed-loop framework that couples the VLM with the VGM at step-level critiques.<n>On Gen-ViRe and VBVR-Bench, CollabVR improves both open-source and closed-source VGMs over single-inference, Pass@$k$, and prior test-time scaling baselines at matched compute.
  arXiv  Detail & Related papers  (2026-05-09T06:39:17Z)
• Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance [65.36257254806647]
  Vision-Language Models (VLMs) have enhanced traditional LLMs with visual capabilities through the integration of vision encoders.<n>We show that common practices, such as choosing encoders with the largest size or highest zero-shot accuracy, consistently fail to identify optimal models.<n>This intriguing finding begs a fundamental question: What factors of vision-encoders matter in VLM?
  arXiv  Detail & Related papers  (2026-05-02T08:42:13Z)
• VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models [43.09726338623949]
  Vision-Language-Action (VLA) models integrate pretrained large Vision-Language Models (VLM) into their policy backbone.<n>This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance.<n>We introduce VLM4VLA, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters.
  arXiv  Detail & Related papers  (2026-01-06T09:58:24Z)
• A Survey on Video Temporal Grounding with Multimodal Large Language Model [107.24431595873808]
  Recent advancement in temporal grounding (VTG) has significantly enhanced fine-grained video understanding.<n>With superior multimodal comprehension and reasoning abilities, VTG approaches based on MLLMs (VTG-MLLMs) are gradually surpassing traditional fine-tuned methods.<n>Despite extensive surveys on general video-language understanding, comprehensive reviews specifically addressing VTG-MLLMs remain scarce.
  arXiv  Detail & Related papers  (2025-08-07T08:52:11Z)
• ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning [68.76048244253582]
  We introduce ViaRL, the first framework to leverage rule-based reinforcement learning (RL) for optimizing frame selection in video understanding.<n>ViaRL utilizes the answer accuracy of a downstream model as a reward signal to train a frame selector through trial-and-error.<n>ViaRL consistently delivers superior temporal grounding performance and robust generalization across diverse video understanding tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:29:40Z)
• VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models [20.92507667350599]
  We introduce a verbalized learning framework named VERA that enables vision-language models to perform video anomaly detection.<n> VERA decomposes the complex reasoning required for VAD into reflections on simpler, more focused guiding questions.<n>During inference, VERA embeds the learned questions into model prompts to guide VLMs in generating segment-level anomaly scores.
  arXiv  Detail & Related papers  (2024-12-02T04:10:14Z)
• Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs [83.24033574914425]
  We present Prism, an innovative framework designed to disentangle the perception and reasoning processes involved in visual question solving. Prism comprises two distinct stages: a perception stage that utilizes a VLM to extract and articulate visual information in textual form, and a reasoning stage that formulates responses based on the extracted visual information. Our analytical framework provides several valuable insights, underscoring Prism's potential as a cost-effective solution for vision-language tasks.
  arXiv  Detail & Related papers  (2024-06-20T17:54:03Z)
• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning [79.38140606606126]
  We propose an algorithmic framework that fine-tunes vision-language models (VLMs) with reinforcement learning (RL) Our framework provides a task description and then prompts the VLM to generate chain-of-thought (CoT) reasoning. We demonstrate that our proposed framework enhances the decision-making capabilities of VLM agents across various tasks.
  arXiv  Detail & Related papers  (2024-05-16T17:50:19Z)
• Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions [126.3136109870403]
  We introduce a generic and lightweight Visual Prompt Generator Complete module (VPG-C) VPG-C infers and completes the missing details essential for comprehending demonstrative instructions. We build DEMON, a comprehensive benchmark for demonstrative instruction understanding.
  arXiv  Detail & Related papers  (2023-08-08T09:32:43Z)
• Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity [45.86789047206224]
  This paper presents novel benchmarks for evaluating vision-language models (VLMs) in zero-shot recognition. Our benchmarks test VLMs' consistency in understanding concepts across semantic granularity levels and their response to varying text specificity. Findings show that VLMs favor moderately fine-grained concepts and struggle with specificity, often misjudging texts that differ from their training data.
  arXiv  Detail & Related papers  (2023-06-28T09:29:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.