Coverage-Driven Adaptive Keyframe Selection for Video Understanding

• URL: http://arxiv.org/abs/2608.00714v1
• Date: Sat, 01 Aug 2026 15:24:06 GMT
• Title: Coverage-Driven Adaptive Keyframe Selection for Video Understanding
• Abstract summary: We propose CSES, a training-free semantic selector that adaptively determines the numbers of frames to score and select.<n> CSES estimates the prominence of the frame-query to guide active acquisition and adapt the temporal coverage of each input.<n>It then formulates selection relevance as a coverage problem that jointly accounts for semantic relevance, temporal redundancy, and visual redundancy.
• Score: 12.078063042214573
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in large vision-language models (LVLMs) have enabled long-video understanding and analysis. However, processing the large number of frames in a video incurs substantial computational overhead. Existing methods reduce LVLM inference costs by scoring frame-query relevance before inference and selecting keyframes accordingly. Nevertheless, the distribution of relevant frames varies across queries, and these methods often need to score hundreds or thousands of frames. To address this limitation, we propose CSES, a training-free semantic keyframe selector that adaptively determines the numbers of frames to score and keyframes to select. CSES estimates the prominence of the frame-query relevance profile to guide active acquisition and adapt the temporal coverage of each input. It then formulates keyframe selection as a coverage problem that jointly accounts for semantic relevance, temporal redundancy, and visual redundancy. Active acquisition and keyframe selection terminate based on coverage saturation. The selection objective is monotone and submodular, enabling greedy optimization with a standard approximation guarantee. Experiments with four LVLMs on two benchmarks show that our method preserves accuracy while scoring $4$-$13\times$ fewer frames and selecting $18.4\%$-$20.5\%$ fewer input keyframes than existing baselines. CSES further achieves a $3.1$-$5.4\times$ speedup in frame selection over baselines.

Related papers

• MarKey: Marginal Utility Guided Greedy Keyframe Selection for Long Video Understanding [17.82627304498428]
  Long-video understanding remains challenging for large language models.<n>Recent training-free selection methods have enabled more efficient inference.<n>We propose MarKey, a framework that formulates multimodal selection as subset-aware greedy optimization.
  arXiv  Detail & Related papers  (2026-09-14T11:39:20Z)
• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding [55.75126327155788]
  RIDGE is a frame selection framework that reads the frame-query similarity curve as a temporal signal.<n>It partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget.<n>It achieves the best performance in most settings and remains competitive in the others.
  arXiv  Detail & Related papers  (2026-08-30T18:40:51Z)
• Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors [69.75895255395702]
  Cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation.<n>A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation.<n>Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets.
  arXiv  Detail & Related papers  (2026-07-17T07:04:31Z)
• QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding [78.36929439152566]
  We propose a Query- and Content-Aware (QCA) selection framework that can select a compact yet information-rich set of frames from long videos.<n>QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query.<n>Our method requires no additional training and can be seamlessly integrated into existing Video-LLMs.
  arXiv  Detail & Related papers  (2026-07-01T14:19:24Z)
• GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding [63.62445065977448]
  GIFT: Global Irreplaceability Frame Targeting is a training-free framework that selects frames by assessing their intrinsic irreplaceability.<n>We show that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling.
  arXiv  Detail & Related papers  (2026-03-26T06:21:41Z)
• Adaptive Greedy Frame Selection for Long Video Understanding [11.923839324117674]
  Large vision--language models (VLMs) are increasingly applied to long-video question answering.<n>Inference is often bottlenecked by the number of input frames and resulting visual tokens.<n>We propose a question-adaptive greedy frame selection method that jointly optimize query relevance and semantic representativeness under a fixed frame budget.
  arXiv  Detail & Related papers  (2026-03-20T17:55:32Z)
• FOCUS: Efficient Keyframe Selection for Long Video Understanding [26.44459739499484]
  Multimodal large language models (LMLMs) represent images and video frames as visual tokens.<n> FOCUS, Frame-Optimistic Confidence Upperbound Selection, is a model-agnostic selection module that selects frames under a strict token budget.<n>For videos longer than 20 minutes, it achieves an 11.9% gain in accuracy on LongVideoBenching benchmarks.
  arXiv  Detail & Related papers  (2025-10-31T08:41:13Z)
• K-frames: Scene-Driven Any-k Keyframe Selection for long video understanding [38.06179287702453]
  K-frames is a novel paradigm for scene-driven selection that preserves temporal continuity.<n>Instead of selecting individual frames, K-frames predicts semantically coherent, query-relevant clips.<n>K-frames provides an effective, interpretable, and plug-and-play solution for selection at various scales.
  arXiv  Detail & Related papers  (2025-10-14T06:23:22Z)
• A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering [15.220013605396396]
  A.I.R. is a training-free approach for Adaptive, Iterative, and Reasoning-based frame selection.<n>We leverage a powerful Vision-Language Models (VLMs) to perform deep, semantic analysis on complex queries.<n>Our approach significantly boosts the performance of the foundation VLM, and achieves substantial gains in computational efficiency.
  arXiv  Detail & Related papers  (2025-10-06T01:51:13Z)
• ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding [52.050036778325094]
  We introduce ReFoCUS (Reinforcement-guided Frame Optimization for Contextual UnderStanding), a novel frame-level policy optimization framework.<n>ReFoCUS learns a frame selection policy via reinforcement learning, using reward signals derived from a reference LMM to reflect the model's intrinsic preferences for frames.<n>Our approach consistently improves reasoning performance across multiple video QA benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T03:08:07Z)
• Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders [62.58375366359421]
  Multimodal Large Language Models (MLLMs) for long video understanding remains a challenging problem.<n>Traditional uniform sampling leads to selection of irrelevant content.<n>Post-training MLLMs on thousands of frames imposes a substantial computational burden.<n>We propose threadings with narratives (Nar-KFC) to facilitate effective and efficient long video perception.
  arXiv  Detail & Related papers  (2025-05-30T03:04:28Z)
• Adaptive Keyframe Sampling for Long Video Understanding [75.7837692594814]
  This paper presents a simple yet effective algorithm named Adaptive Keyframe Sampling (AKS)<n>It inserts a plug-and-play module known as Adaptive Keyframe Sampling (AKS) which aims to maximize the useful information with a fixed number of video tokens.<n>Experiments on two long video understanding benchmarks validate that AKS improves video QA accuracy upon selecting informative encounters.
  arXiv  Detail & Related papers  (2025-02-28T17:46:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.