Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding

• URL: http://arxiv.org/abs/2609.11899v1
• Date: Thu, 10 Sep 2026 17:53:59 GMT
• Title: Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding
• Abstract summary: Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth budgets.<n>We propose Caption-once, Frames-onDemand, a budget-aware edge-cloud agentic framework.<n> Experiments on long-video benchmarks demonstrate strong accuracy-efficiency trade-offs while substantially reducing online visual processing.
• Score: 59.773752487686444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth budgets. Subsampling visual tokens loses temporal structure, while text-only video memories lose fine-grained visual attributes. We observe a visual-textual duality: language memories carry long-range temporal structure better than dense frames, while pixels remain decisive for attribute-level perception. Building on this insight, we propose Caption-once, Frames-onDemand (CFD), a budget-aware edge-cloud agentic framework. The edge runs a single offline captioning pass that builds a dual-track narrative index, an event-level story skeleton plus a clip-level micro-log, cached and reused across queries without re-captioning. At query time, a cloud-side MLLM reasons over the index in a story-first loop centered on a lightweight Visual-Need Router: a per-query gating module that triggers bounded keyframe retrieval only for perceptual questions (appearance, on-screen text, attribute disambiguation) and keeps temporal-structural questions in language space. The router turns visual access into a first-class, query-conditioned cost, capping per-query frame consumption regardless of video length. Experiments on long-video benchmarks demonstrate strong accuracy-efficiency trade-offs while substantially reducing online visual processing.

Related papers

• Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs [2.1665689529884697]
  Video understanding has rapidly evolved toward video large language models (VideoLLMs)<n>Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length.<n>This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count.
  arXiv  Detail & Related papers  (2026-09-09T15:50:39Z)
• Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning [77.7531245219403]
  textscHomer is a Hierarchical Online Memory Exploration and Reasoning framework.<n>textscHomer's memory mirrors the multi-scale structure of long videos, ranging from raw perception, to recurring entities, to events connected by explicit temporal and causal relations.<n>textscHomer outperforms the previous best agent method by $+5.5$, $+10.8$, and $+4.4$ points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long.
  arXiv  Detail & Related papers  (2026-07-01T05:53:09Z)
• Q-Fold: Query-Aware Focus-Context Spatio-Temporal Folding for Long Video Understanding [24.403140075177518]
  We propose Q-Fold, a training-free input construction framework for long-video understanding.<n>Instead of treating isolated frames as the basic modeling unit, Q-Fold operates on contiguous temporal segments.<n>Experiments show that Q-Fold consistently improves performance without increasing the input budget.
  arXiv  Detail & Related papers  (2026-06-10T14:19:15Z)
• Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding [69.296913137409]
  We introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS)<n>We present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations.<n>First, the Frame-Driven Transition Controller (FDTC), makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent.<n>Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content.
  arXiv  Detail & Related papers  (2026-06-05T07:29:20Z)
• ReCA: Multi-Shot Long Video Extrapolation via Recursive Context Allocation [69.45329019089041]
  Multi-Shot Video Extrapolation (MSVE) is a task that extends an observed frame or clip into a sequence of cinematically structured shots.<n>MSVE reveals that long-video failure is not merely a limitation of context length, but a failure of context allocation.<n>We propose Recursive Context Allocation (ReCA), an inference-time framework that allocates context hierarchically across planning and generation.
  arXiv  Detail & Related papers  (2026-05-26T04:16:56Z)
• VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAG [33.938909878793815]
  VideoStir is a structured and multimodal intent-aware long-video RAG framework.<n>It structures a video as atemporal graph at clip level, and then performs multi-hop retrieval to aggregate evidence.<n>It also introduces an MLLM-backed intent-relevance scorer that retrieves frames based on their alignment with the query's reasoning intent.
  arXiv  Detail & Related papers  (2026-04-07T04:26:59Z)
• ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting [17.594941754364484]
  Video Large Language Models (VideoLLMs) have enabled strong performance across diverse multimodal video tasks.<n>To reduce the high computational cost of processing dense video frames, efficiency-oriented methods such as frame selection have been widely adopted.<n>We introduce ViKey, a training-free framework that combines VP with a lightweight Keyword-Frame Mapping (KFM) module.
  arXiv  Detail & Related papers  (2026-03-24T13:32:52Z)
• Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models [61.11154533305096]
  Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens.<n>We propose a new perspective that elaborates token textbfAnchors within intra-frame and inter-frame contexts.<n>Our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs.
  arXiv  Detail & Related papers  (2026-03-02T03:06:40Z)
• TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding [14.570869250170139]
  TV-RAG is a training-free architecture that couples temporal alignment with entropy-guided semantics to improve long-video reasoning.<n>By weaving these temporal and semantic signals together, TV-RAG realises a dual-level reasoning routine that can be grafted onto any LVLM without re-training or fine-tuning.
  arXiv  Detail & Related papers  (2025-12-29T14:10:22Z)
• Episodic Memory Representation for Long-form Video Understanding [52.33907540905242]
  Large Video Language Models excel at general video understanding but struggle with long-form context window limits.<n>We introduce Video-EM, a training free framework inspired by the principles of human memory.<n>Video-EM achieves performance gains of 4-9 percent over respective baselines while utilizing fewer frames.
  arXiv  Detail & Related papers  (2025-08-13T04:33:07Z)
• Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting [60.58915701973593]
  We present CAT-V (Caption AnyThing in Video), a training-free framework for fine-grained object-centric video captioning.<n>Cat-V integrates three key components: a Segmenter based on SAMI for precise object segmentation across frames, a Temporal Analyzer powered by TRACE-UniVL, and a Captioner using Intern-2.5.<n>Our framework generates detailed, temporally-aware descriptions of objects' attributes, actions, statuses, interactions, and environmental contexts without requiring additional training data.
  arXiv  Detail & Related papers  (2025-04-07T22:35:36Z)
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding [65.46303012350207]
  LongVU is an adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. We leverage DINOv2 features to remove redundant frames that exhibit high similarity. We perform spatial token reduction across frames based on their temporal dependencies.
  arXiv  Detail & Related papers  (2024-10-22T21:21:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.