Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors

• URL: http://arxiv.org/abs/2607.15689v1
• Date: Fri, 17 Jul 2026 07:04:31 GMT
• Title: Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors
• Abstract summary: Cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation.<n>A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation.<n>Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets.
• Score: 69.75895255395702
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding long videos with multimodal large language models (MLLMs) requires selecting a compact set of frames from thousands of candidates, yet identifying the right frames seemingly requires understanding the video first. We resolve this circular dependency with a simple observation: cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation. We exploit this property to build DAFS (Dynamic Attention-based Budget-aware Frame Selection), a training-free frame selector. A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation. This enables cross-frame comparison without autoregressive decoding. To handle the selector's own context constraint, we formulate the joint allocation of candidate pool size and per-frame token budget as a discrete optimization problem solved by dynamic programming. Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets, while generalizing across selector and answerer backbones, and across tasks, without retraining.

Related papers

• MarKey: Marginal Utility Guided Greedy Keyframe Selection for Long Video Understanding [17.82627304498428]
  Long-video understanding remains challenging for large language models.<n>Recent training-free selection methods have enabled more efficient inference.<n>We propose MarKey, a framework that formulates multimodal selection as subset-aware greedy optimization.
  arXiv  Detail & Related papers  (2026-09-14T11:39:20Z)
• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding [55.75126327155788]
  RIDGE is a frame selection framework that reads the frame-query similarity curve as a temporal signal.<n>It partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget.<n>It achieves the best performance in most settings and remains competitive in the others.
  arXiv  Detail & Related papers  (2026-08-30T18:40:51Z)
• Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding [54.840531495584095]
  We propose a dynamic visual selection framework named EviSelect grounded in the target MLLM's internal attention evidence.<n>Our method efficiently approximates maps of the target MLLM using highly compressed visual inputs and sparse attention.<n>EviSelect achieves superior performance compared to existing methods while reducing selected visual tokens by about 50% and achieving a 3.9x end-to-end speedup.
  arXiv  Detail & Related papers  (2026-08-06T09:15:43Z)
• Coverage-Driven Adaptive Keyframe Selection for Video Understanding [12.078063042214573]
  We propose CSES, a training-free semantic selector that adaptively determines the numbers of frames to score and select.<n> CSES estimates the prominence of the frame-query to guide active acquisition and adapt the temporal coverage of each input.<n>It then formulates selection relevance as a coverage problem that jointly accounts for semantic relevance, temporal redundancy, and visual redundancy.
  arXiv  Detail & Related papers  (2026-08-01T15:24:06Z)
• Where to Focus: Query-Modulated Multimodal Keyframe Selection for Long Video Understanding [48.14432643308697]
  Long video understanding is a formidable challenge for Multimodal Large Language Models.<n>Q-Gate is a plug-and-play framework that treats selection as a dynamic modality routing problem.
  arXiv  Detail & Related papers  (2026-04-19T13:04:18Z)
• Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding [21.306038832082553]
  Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering.<n>Their application to long-form videos is constrained by limited context length and computational cost.<n>We propose an evidence-driven sampling framework grounded in information theory.
  arXiv  Detail & Related papers  (2026-04-01T15:02:50Z)
• Event-Anchored Frame Selection for Effective Long-Video Understanding [67.56884568828508]
  Event-Anchored Frame Selection (EFS) is a hierarchical, event-aware pipeline.<n>As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs.
  arXiv  Detail & Related papers  (2026-03-01T08:25:37Z)
• Improving Video Question Answering through query-based frame selection [15.416301612152004]
  Video Question Answering (VideoQA) models enhance understanding and interaction with audiovisual content.<n>Due to heavy compute requirements, most large visual language models (VLMs) for VideoQA rely on a fixed number of frames by uniformly sampling the video.<n>We present a novel query-based selection of frames relevant to the questions based on the submodular mutual Information (SMI) functions.
  arXiv  Detail & Related papers  (2026-01-12T12:10:20Z)
• HFS: Holistic Query-Aware Frame Selection for Efficient Video Reasoning [13.569944737211472]
  Key frame selection in video understanding presents significant challenges.<n>Traditional top-K selection methods, which score frames independently, often fail to optimize the selection as a whole.<n>We propose an end-to-end trainable, task-adaptive framework for frame selection.
  arXiv  Detail & Related papers  (2025-12-12T13:10:30Z)
• Adaptive Keyframe Sampling for Long Video Understanding [75.7837692594814]
  This paper presents a simple yet effective algorithm named Adaptive Keyframe Sampling (AKS)<n>It inserts a plug-and-play module known as Adaptive Keyframe Sampling (AKS) which aims to maximize the useful information with a fixed number of video tokens.<n>Experiments on two long video understanding benchmarks validate that AKS improves video QA accuracy upon selecting informative encounters.
  arXiv  Detail & Related papers  (2025-02-28T17:46:29Z)
• M-LLM Based Video Frame Selection for Efficient Video Understanding [60.93714759178143]
  We propose a light-weight M-LLM-based frame selection method that adaptively select frames that are more relevant to users' queries.<n>The selected frames are then digested by a frozen downstream video M-LLM for visual reasoning and question answering.
  arXiv  Detail & Related papers  (2025-02-27T01:44:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.