One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding

• URL: http://arxiv.org/abs/2608.05707v1
• Date: Thu, 06 Aug 2026 07:47:15 GMT
• Title: One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding
• Abstract summary: We introduce Matryoshka Evidence-to-Context (MEC) Frame Selection, a training-free framework that builds a reusable sparse video index.<n>MEC improves average accuracy over uniform sampling by 3.77 percentage points, matches strong state-of-the-art selectors, and reduces end-to-end selection latency by 47.37-51.19%.
• Score: 43.953366792790625
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Frame selection is essential for applying Large Multimodal Models (LMMs) to long videos due to severe frame redundancy and limited context windows. Since the appropriate frame budget varies with the downstream LMM, reasoning demands, and latency constraints, a practical selector should serve multiple budgets. However, existing methods typically optimize an isolated frame subset for each predefined budget: when the budget changes, previously selected evidence may be replaced rather than progressively augmented. Ranking frames by a fixed score would allow prefix reuse across budgets, but it ignores the distinct roles of different ranking positions. In this paper, we formulate long-video frame selection as a Matryoshka ranking problem: constructing a single priority sequence whose small prefixes concentrate query-conditioned evidence, while progressively larger prefixes preserve this evidence and add broader temporal context. Efficiently constructing such a ranking is itself challenging, as densely sampling long videos and evaluating frame-query relevance incurs substantial overhead. We therefore introduce Matryoshka Evidence-to-Context (MEC) Frame Selection, a training-free framework that builds a reusable sparse video index, discovers candidates through sparse probing and local zooming, and greedily constructs a position-adaptive ranking: early positions emphasize evidence; later positions progressively favor temporal coverage while preserving visual diversity. A single ranking can thus be truncated to any target budget without rerunning the selector. Across four benchmarks and six frame budgets, MEC improves average accuracy over uniform sampling by 3.77 percentage points, matches strong state-of-the-art selectors, and reduces end-to-end selection latency by 47.37-51.19%.

Related papers

• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding [55.75126327155788]
  RIDGE is a frame selection framework that reads the frame-query similarity curve as a temporal signal.<n>It partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget.<n>It achieves the best performance in most settings and remains competitive in the others.
  arXiv  Detail & Related papers  (2026-08-30T18:40:51Z)
• When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding [17.062781672214104]
  We propose EcoFrame, a framework for low-overhead query-adaptive visual evidence scheduling.<n>Experiments on Video-MME, LongVideoBench, and MLVU demonstrate that EcoFrame achieves a better accuracy--efficiency trade-off across multiple VLM backbones.
  arXiv  Detail & Related papers  (2026-08-04T16:49:53Z)
• Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering [9.937860041147625]
  Long-video question answering requires identifying sparse yet critical evidence from videos under a constrained visual-token budget.<n>Existing methods either select query-aware frames in a single pass or rely on timestamped text solely as retrieval guidance.<n>We propose GCR, a training-free framework that casts fixed-budget frame selection as a joint evidence curation problem.
  arXiv  Detail & Related papers  (2026-08-03T03:50:22Z)
• Coverage-Driven Adaptive Keyframe Selection for Video Understanding [12.078063042214573]
  We propose CSES, a training-free semantic selector that adaptively determines the numbers of frames to score and select.<n> CSES estimates the prominence of the frame-query to guide active acquisition and adapt the temporal coverage of each input.<n>It then formulates selection relevance as a coverage problem that jointly accounts for semantic relevance, temporal redundancy, and visual redundancy.
  arXiv  Detail & Related papers  (2026-08-01T15:24:06Z)
• Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors [69.75895255395702]
  Cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation.<n>A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation.<n>Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets.
  arXiv  Detail & Related papers  (2026-07-17T07:04:31Z)
• QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding [78.36929439152566]
  We propose a Query- and Content-Aware (QCA) selection framework that can select a compact yet information-rich set of frames from long videos.<n>QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query.<n>Our method requires no additional training and can be seamlessly integrated into existing Video-LLMs.
  arXiv  Detail & Related papers  (2026-07-01T14:19:24Z)
• GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding [63.62445065977448]
  GIFT: Global Irreplaceability Frame Targeting is a training-free framework that selects frames by assessing their intrinsic irreplaceability.<n>We show that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling.
  arXiv  Detail & Related papers  (2026-03-26T06:21:41Z)
• Adaptive Greedy Frame Selection for Long Video Understanding [11.923839324117674]
  Large vision--language models (VLMs) are increasingly applied to long-video question answering.<n>Inference is often bottlenecked by the number of input frames and resulting visual tokens.<n>We propose a question-adaptive greedy frame selection method that jointly optimize query relevance and semantic representativeness under a fixed frame budget.
  arXiv  Detail & Related papers  (2026-03-20T17:55:32Z)
• Event-Anchored Frame Selection for Effective Long-Video Understanding [67.56884568828508]
  Event-Anchored Frame Selection (EFS) is a hierarchical, event-aware pipeline.<n>As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs.
  arXiv  Detail & Related papers  (2026-03-01T08:25:37Z)
• ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding [52.050036778325094]
  We introduce ReFoCUS (Reinforcement-guided Frame Optimization for Contextual UnderStanding), a novel frame-level policy optimization framework.<n>ReFoCUS learns a frame selection policy via reinforcement learning, using reward signals derived from a reference LMM to reflect the model's intrinsic preferences for frames.<n>Our approach consistently improves reasoning performance across multiple video QA benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T03:08:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.