Adaptive Greedy Frame Selection for Long Video Understanding

• URL: http://arxiv.org/abs/2603.20180v1
• Date: Fri, 20 Mar 2026 17:55:32 GMT
• Title: Adaptive Greedy Frame Selection for Long Video Understanding
• Abstract summary: Large vision--language models (VLMs) are increasingly applied to long-video question answering.<n>Inference is often bottlenecked by the number of input frames and resulting visual tokens.<n>We propose a question-adaptive greedy frame selection method that jointly optimize query relevance and semantic representativeness under a fixed frame budget.
• Score: 11.923839324117674
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large vision--language models (VLMs) are increasingly applied to long-video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweight text-only question-type classifier that routes each query to its best-performing preset. Experiments on MLVU show consistent accuracy gains over uniform sampling and a strong recent baseline across frame budgets, with the largest improvements under tight budgets.

Related papers

• MarKey: Marginal Utility Guided Greedy Keyframe Selection for Long Video Understanding [17.82627304498428]
  Long-video understanding remains challenging for large language models.<n>Recent training-free selection methods have enabled more efficient inference.<n>We propose MarKey, a framework that formulates multimodal selection as subset-aware greedy optimization.
  arXiv  Detail & Related papers  (2026-09-14T11:39:20Z)
• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding [55.75126327155788]
  RIDGE is a frame selection framework that reads the frame-query similarity curve as a temporal signal.<n>It partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget.<n>It achieves the best performance in most settings and remains competitive in the others.
  arXiv  Detail & Related papers  (2026-08-30T18:40:51Z)
• One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding [43.953366792790625]
  We introduce Matryoshka Evidence-to-Context (MEC) Frame Selection, a training-free framework that builds a reusable sparse video index.<n>MEC improves average accuracy over uniform sampling by 3.77 percentage points, matches strong state-of-the-art selectors, and reduces end-to-end selection latency by 47.37-51.19%.
  arXiv  Detail & Related papers  (2026-08-06T07:47:15Z)
• Coverage-Driven Adaptive Keyframe Selection for Video Understanding [12.078063042214573]
  We propose CSES, a training-free semantic selector that adaptively determines the numbers of frames to score and select.<n> CSES estimates the prominence of the frame-query to guide active acquisition and adapt the temporal coverage of each input.<n>It then formulates selection relevance as a coverage problem that jointly accounts for semantic relevance, temporal redundancy, and visual redundancy.
  arXiv  Detail & Related papers  (2026-08-01T15:24:06Z)
• Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors [69.75895255395702]
  Cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation.<n>A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation.<n>Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets.
  arXiv  Detail & Related papers  (2026-07-17T07:04:31Z)
• Gaussian Mixture Modeling for Event-Aware Visual Allocation in Long Video Understanding [55.04091863981529]
  Large Vision-Language Models (LVLMs) face significant challenges in long video understanding due to the excessive computational cost and information loss associated with uniform sampling.<n>We propose GMM-EVA, which leverages Gaussian Mixture Models to model event-level structure from discrete frame-wise.<n>A differentiated allocation strategy is then applied to preserve one primary high-resolution per event for high-fidelity detail, while utilizing lower-resolution secondary observations to maintain temporal context and optimize token budgets.
  arXiv  Detail & Related papers  (2026-07-14T09:27:25Z)
• GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding [63.62445065977448]
  GIFT: Global Irreplaceability Frame Targeting is a training-free framework that selects frames by assessing their intrinsic irreplaceability.<n>We show that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling.
  arXiv  Detail & Related papers  (2026-03-26T06:21:41Z)
• Event-Anchored Frame Selection for Effective Long-Video Understanding [67.56884568828508]
  Event-Anchored Frame Selection (EFS) is a hierarchical, event-aware pipeline.<n>As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs.
  arXiv  Detail & Related papers  (2026-03-01T08:25:37Z)
• Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding [43.587729230845525]
  Current methods typically select frames with high relevance to a given query.<n>We introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework.<n>WFS-SB significantly boosts LVLM performance, improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench.
  arXiv  Detail & Related papers  (2026-02-28T07:18:07Z)
• Improving Video Question Answering through query-based frame selection [15.416301612152004]
  Video Question Answering (VideoQA) models enhance understanding and interaction with audiovisual content.<n>Due to heavy compute requirements, most large visual language models (VLMs) for VideoQA rely on a fixed number of frames by uniformly sampling the video.<n>We present a novel query-based selection of frames relevant to the questions based on the submodular mutual Information (SMI) functions.
  arXiv  Detail & Related papers  (2026-01-12T12:10:20Z)
• FOCUS: Efficient Keyframe Selection for Long Video Understanding [26.44459739499484]
  Multimodal large language models (LMLMs) represent images and video frames as visual tokens.<n> FOCUS, Frame-Optimistic Confidence Upperbound Selection, is a model-agnostic selection module that selects frames under a strict token budget.<n>For videos longer than 20 minutes, it achieves an 11.9% gain in accuracy on LongVideoBenching benchmarks.
  arXiv  Detail & Related papers  (2025-10-31T08:41:13Z)
• K-frames: Scene-Driven Any-k Keyframe Selection for long video understanding [38.06179287702453]
  K-frames is a novel paradigm for scene-driven selection that preserves temporal continuity.<n>Instead of selecting individual frames, K-frames predicts semantically coherent, query-relevant clips.<n>K-frames provides an effective, interpretable, and plug-and-play solution for selection at various scales.
  arXiv  Detail & Related papers  (2025-10-14T06:23:22Z)
• LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoning [73.90466023069125]
  We propose LOVE-R1, a model that can adaptively zoom in on a video clip.<n>The model is first provided with densely sampled frames but in a small resolution.<n>If some spatial details are needed, the model can zoom in on a clip of interest with a large frame resolution.
  arXiv  Detail & Related papers  (2025-09-29T13:43:55Z)
• ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding [52.050036778325094]
  We introduce ReFoCUS (Reinforcement-guided Frame Optimization for Contextual UnderStanding), a novel frame-level policy optimization framework.<n>ReFoCUS learns a frame selection policy via reinforcement learning, using reward signals derived from a reference LMM to reflect the model's intrinsic preferences for frames.<n>Our approach consistently improves reasoning performance across multiple video QA benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T03:08:07Z)
• MHSCNet: A Multimodal Hierarchical Shot-aware Convolutional Network for Video Summarization [61.69587867308656]
  We propose a multimodal hierarchical shot-aware convolutional network, denoted as MHSCNet, to enhance the frame-wise representation. Based on the learned shot-aware representations, MHSCNet can predict the frame-level importance score in the local and global view of the video.
  arXiv  Detail & Related papers  (2022-04-18T14:53:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.