LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents

• URL: http://arxiv.org/abs/2608.07585v1
• Date: Wed, 05 Aug 2026 10:25:47 GMT
• Title: LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents
• Abstract summary: Long-video understanding requires models to efficiently acquire and reuse sparse visual evidence from long and redundant video streams.<n>Recent video tool-use agents address this challenge by iteratively invoking visual Tools at different temporal scales.<n>We propose Latent Visual Evidence-Enhanced Planning (LAVE), a training-free framework for reusing latent visual evidence from completed Tool calls.
• Score: 31.505673981223385
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-video understanding requires models to efficiently acquire and reuse sparse visual evidence from long and redundant video streams. Recent video tool-use agents address this challenge by iteratively invoking visual Tools at different temporal scales, but their Tool-Planner communication typically relies on textual observations. Such text-only interfaces provide lossy summaries of Tool computations, causing previously computed visual evidence not verbalized to be discarded and unavailable for subsequent planning. We identify this limitation as the Tool observation bottleneck and propose Latent Visual Evidence-Enhanced Planning (LAVE), a training-free framework for reusing latent visual evidence from completed Tool calls. LAVE introduces a dual-channel observation interface: the visible channel preserves the original textual trajectory, while the latent channel stores pre-verbal visual updates with their Tool roles, source-frame timestamps, and visual locations. During planning, LAVE retrieves evidence relevant to the current Planner state but not covered by textual observations, and integrates it through bounded timestamp-aligned latent updates with entropy-constrained frame-time routing. This enables video agents to reuse existing visual computation without additional training, frame replay, or modifications to the original orchestration. Extensive experiments on Video-MME, LongVideoBench, and CG-Bench show that LAVE consistently improves video tool-use agents across backbones. Under a comparable frame budget, LAVE improves the Video-MME overall score by 3.76 points over the strongest baseline, demonstrating the effectiveness of latent visual evidence reuse for multi-step video-agent planning.

Related papers

• Visual Search Augmented Chain-of-Thought Reasoning for Attribute Value Extraction from Product Videos [14.43540077402288]
  We propose visual search augmented chain-of-thought reasoning (ViS-CoT) for visual attribute value extraction.<n>ViS-CoT can be easily applied to any open-source video VLM for video-to-text AVE in e-Commerce.
  arXiv  Detail & Related papers  (2026-09-06T06:01:57Z)
• STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models [81.32710031596591]
  Video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames.<n>Existing methods built on vision-language models (LVLMs) often address this challenge by externalizing reasoning through chain-of-thought (CoT)<n>We propose STORMS, a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT.
  arXiv  Detail & Related papers  (2026-05-25T16:33:00Z)
• ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding [30.50286834537385]
  Video understanding requires active evidence seeking, motivating tool-augmented video agents for temporal reasoning, cross-modal understanding, and complex question answering.<n>Existing video agents have improved video reasoning with retrieval, memory, frame inspection, and verifier tools, but they still face two limitations.<n>In this paper, we address these challenges with two complementary designs.
  arXiv  Detail & Related papers  (2026-05-13T09:19:22Z)
• Act2See: Emergent Active Visual Perception for Video Reasoning [70.52373240573856]
  Vision-Language Models (VLMs) typically rely on static initial frames for video reasoning.<n>We introduce Act-to-See (Act2See), a novel framework that enables active visual perception.
  arXiv  Detail & Related papers  (2026-05-03T00:52:51Z)
• APVR: Hour-Level Long Video Understanding with Adaptive Pivot Visual Information Retrieval [41.81696346270799]
  Current large language models (LMs) struggle with hour-level video understanding.<n>bftextAdaptive textbfPivot MLbfVisual information textbfRetrieval (textbfAPVR), a training-free framework that hierarchically retrieves and retains sufficient and important visual information.
  arXiv  Detail & Related papers  (2025-06-05T12:27:10Z)
• Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding [60.88843818016968]
  Long-form video understanding presents significant challenges due to temporal-spatial complexity and difficulty of question answering.<n>We propose the Deep Video Discovery (DVD) agent to leverage an agentic search strategy over segmented video clips.<n>Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of 74.2%.
  arXiv  Detail & Related papers  (2025-05-23T16:37:36Z)
• Scoring, Remember, and Reference: Catching Camouflaged Objects in Videos [24.03405963900272]
  Video Camouflaged Object Detection aims to segment objects whose appearances closely resemble their surroundings.<n>Existing vision models often struggle in such scenarios due to the indistinguishable appearance of camouflaged objects.<n>We propose an end-to-end framework inspired by human memory-recognition.
  arXiv  Detail & Related papers  (2025-03-21T11:08:14Z)
• Understanding Long Videos via LLM-Powered Entity Relation Graphs [51.13422967711056]
  GraphVideoAgent is a framework that maps and monitors the evolving relationships between visual entities throughout the video sequence.<n>Our approach demonstrates remarkable effectiveness when tested against industry benchmarks.
  arXiv  Detail & Related papers  (2025-01-27T10:57:24Z)
• Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biases [69.46487306858789]
  Conditional Autoregressive Slot Attention (CA-SA) is a framework that enhances the temporal consistency of extracted object-centric representations in video-centric vision tasks. We present qualitative and quantitative results showing that our proposed method outperforms the considered baselines on downstream tasks.
  arXiv  Detail & Related papers  (2024-10-21T07:44:44Z)
• HAVANA: Hierarchical stochastic neighbor embedding for Accelerated Video ANnotAtions [59.71751978599567]
  This paper presents a novel annotation pipeline that uses pre-extracted features and dimensionality reduction to accelerate the temporal video annotation process. We demonstrate significant improvements in annotation effort compared to traditional linear methods, achieving more than a 10x reduction in clicks required for annotating over 12 hours of video.
  arXiv  Detail & Related papers  (2024-09-16T18:15:38Z)
• Video-based Person Re-identification with Long Short-Term Representation Learning [101.62570747820541]
  Video-based person Re-Identification (V-ReID) aims to retrieve specific persons from raw videos captured by non-overlapped cameras. We propose a novel deep learning framework named Long Short-Term Representation Learning (LSTRL) for effective V-ReID.
  arXiv  Detail & Related papers  (2023-08-07T16:22:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.