Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA
- URL: http://arxiv.org/abs/2607.16189v1
- Date: Fri, 17 Jul 2026 17:59:27 GMT
- Title: Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA
- Abstract summary: Grounded long-video question answering (Grounded LVQA) requires answering a question about a long video while localizing the short evidence interval that supports the answer.<n>Recent agentic methods frame this task as multi-turn exploration with a single crop_video(start, end) action, which supports coarse-to-fine narrowing.<n>We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree.
- Score: 64.65523062099061
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Grounded long-video question answering (Grounded LVQA) requires answering a question about a long video while localizing the short evidence interval that supports the answer. Recent agentic methods frame this task as multi-turn exploration with a single crop_video(start, end) action, which supports coarse-to-fine narrowing but provides no primitive for fine-to-coarse backtracking. As a result, these agents typically converge prematurely and cannot recover from an early mistake. We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree. VTS constructs a non-uniform tree from visual scene boundaries so that each node corresponds to a semantically coherent segment, and trains an agent to navigate the tree through four discrete operations: zoom_in, zoom_out, shift, and answer. These operations expose backtracking and recovery as explicit, learnable primitives rather than implicit behaviors. To train this navigation, we introduce a trajectory synthesis pipeline that produces multi-step paths through the tree, including deliberate detours into incorrect branches followed by recovery. We use these trajectories for supervised fine-tuning, followed by reinforcement learning with grounding and answer-accuracy rewards. On three Grounded LVQA benchmarks (CG-Bench, Haystack-LVBench, Haystack-Ego4D), VTS outperforms the strongest prior agentic methods by +12.5 mIoU on CG-Bench and +7.4 T-F1 on Haystack-Ego4D. The learned policy also transfers to general long-video QA, surpassing all prior agentic baselines on Video-MME, MLVU, and LVBench by up to +7.1 accuracy points. Ablations confirm that self-correcting hierarchical search is the central mechanism behind these gains: removing either adaptive descent or explicit backtracking substantially degrades performance. Code is available at https://github.com/CeeZh/VTS.
Related papers
- Evidence-Backed Video Question Answering [90.93225758130426]
Existing explainability efforts rely on textual rationales or sparse bounding boxes.<n>We propose Evidence-Backed Video Question Answering (E-VQA)<n>To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding.
arXiv Detail & Related papers (2026-07-13T17:49:10Z) - VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authority [10.961731621875918]
Long video question answering requires locating sparse, time-inspected visual evidence within highly redundant content.<n>We show that existing LVU agents can exhibit "evidence misalignment"<n>We propose the decoupled planner-or framework, which separates planning from answer authority and gates final answering on pixel-level verification.
arXiv Detail & Related papers (2026-05-12T10:37:49Z) - LongVideoAgent: Multi-Agent Reasoning with Long Videos [69.28914905197426]
We propose a multi-agent framework in which a master LLM coordinates a grounding agent to localize question-relevant segments and a vision agent to extract targeted textual observations.<n>The master agent plans with a step limit, and is trained with reinforcement learning to encourage concise, correct, and efficient multi-agent cooperation.<n>On our proposed LongTVQA and LongTVQA+ which are episode-level datasets aggregated from TVQA/TVQA+, our multi-agent system significantly outperforms strong non-agent baselines.
arXiv Detail & Related papers (2025-12-23T18:59:49Z) - MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering [64.46361702927457]
Grounded Video Question Answering (Grounded VideoQA) requires aligning textual answers with explicit visual evidence.<n>In this work, we propose MUPA, a cooperative MUlti-Path Agentic approach that unifies video grounding, question answering, answer reflection and aggregation.
arXiv Detail & Related papers (2025-06-22T15:39:02Z) - VideoExplorer: Think With Videos For Agentic Long-Video Understanding [117.68219930263153]
Long-video understanding is a challenging problem in computer vision.<n>We propose VideoExplorer, a framework grounded in the principle of thinking with video''<n>Rather than reasoning over a static context, VideoExplorer iteratively formulates sub-questions, locates relevant moments, and performs task-oriented, temporally scalable video understanding.
arXiv Detail & Related papers (2025-06-12T15:39:10Z) - ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validation [49.1574468325115]
This work presents an LLM-brained agent for zero-shot Video Question Answering (VideoQA)<n>It combines a Chain-of-Thought framework with grounding reasoning alongside YOLO-World to enhance object tracking and alignment.<n>This approach establishes a new state-of-the-art in VideoQA and Video Understanding, showing enhanced performance on NExT-QA, iVQA, and ActivityNet-QA benchmarks.
arXiv Detail & Related papers (2025-05-21T18:32:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.