Incentivizing Vision Language Models to Search for Long Video Question Answering

• URL: http://arxiv.org/abs/2607.02959v1
• Date: Fri, 03 Jul 2026 05:05:59 GMT
• Title: Incentivizing Vision Language Models to Search for Long Video Question Answering
• Abstract summary: VSeek transforms long-video question answering (LVQA) into a multi-turn retrieval process.<n>We propose a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification.<n>VSeek improves Pass@1 scores by up to 8% and Pass@4 scores by 15% on long-video understanding benchmarks.
• Score: 41.69519333811218
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process. VSeek utilizes a natural language-driven search to identify relevant context within long videos and is post-trained with reinforcement learning (RL) to jointly formulate targeted search queries and reason over retrieved clips for LVQA. While RL post-training has revolutionized reasoning in symbolic domains such as mathematics and code, its application to long-video understanding remains hindered by a lack of verified rewards. To ensure that the retrieved context is relevant, we propose a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification. Specifically, complex user queries are compiled into formal temporal logic specifications for systematically decomposing natural language questions into a definitive checklist of required atomic visual primitives, such as key objects and activities, along with their temporal ordering. These systematically derived grounding events provide the critical feedback signal for RL post-training, enabling dense, verifiable rewards based on the successful retrieval of these specific visual elements rather than relying entirely on outcome-only answer accuracy. By explicitly optimizing for this verifiable evidence-seeking behavior, VSeek improves Pass@1 scores by up to 8% and Pass@4 scores by 15% on long-video understanding benchmarks compared to base models. We open-source our code at https://utaustin-swarmlab.github.io/VSeek.

Related papers

• ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning [78.94047086588078]
  We propose Recursive Evidence as LLM Harness for Long-Context Reasoning.<n>It is a training-free inference method for improving long-context reasoning.<n> Experiments on eight long-context datasets with 128K context length show it consistently improves evidence utilization.
  arXiv  Detail & Related papers  (2026-07-02T17:59:26Z)
• CACR:Reinforcing Temporal Answer Grounding in Instructional Video via Candidate-Aware Causal Reasoning [8.230031783815749]
  TAGV aims to locate precise video segments that respond to natural language queries.<n>This task remains challenging due to the need to comprehend semantically complex questions.<n>We propose a Candidate-Aware Causal Reasoning framework to tackle these limitations.
  arXiv  Detail & Related papers  (2026-06-07T03:23:18Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding [56.45689495743107]
  Vgent is a graph-based retrieval-reasoning-augmented generation framework to enhance LVLMs for long video understanding.<n>We evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-15T19:14:58Z)
• NeuS-QA: Grounding Long-Form Video Understanding in Temporal Logic and Neuro-Symbolic Reasoning [25.109179044490844]
  Long-Form Question Answering (LVQA) poses challenges beyond traditional visual question answering (VQA)<n> Vanilla approaches, which sample frames uniformly and feed them to a VLM with the question, incur significant token overhead.<n>NeuS-QA translates a natural language question into a formal temporal logic expression, constructs a video automaton from frame-level semantic propositions.
  arXiv  Detail & Related papers  (2025-09-22T17:15:13Z)
• VideoExplorer: Think With Videos For Agentic Long-Video Understanding [117.68219930263153]
  Long-video understanding is a challenging problem in computer vision.<n>We propose VideoExplorer, a framework grounded in the principle of thinking with video''<n>Rather than reasoning over a static context, VideoExplorer iteratively formulates sub-questions, locates relevant moments, and performs task-oriented, temporally scalable video understanding.
  arXiv  Detail & Related papers  (2025-06-12T15:39:10Z)
• QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension [86.0749609778104]
  We propose QuoTA, an ante-hoc training-free modular that extends existing large video-language models.<n>QuoTA strategically allocates frame-level importance scores based on query relevance.<n>We decouple the query through Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame importance scoring.
  arXiv  Detail & Related papers  (2025-03-11T17:59:57Z)
• Semantic-aware Dynamic Retrospective-Prospective Reasoning for Event-level Video Question Answering [14.659023742381777]
  Event-Level Video Question Answering (EVQA) requires complex reasoning across video events to provide optimal answers. We propose a semantic-aware dynamic retrospective-prospective reasoning approach for video-based question answering. Our proposed approach achieves superior performance compared to previous state-of-the-art models.
  arXiv  Detail & Related papers  (2023-05-14T03:57:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.