FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding

• URL: http://arxiv.org/abs/2607.25266v2
• Date: Thu, 30 Jul 2026 21:59:18 GMT
• Title: FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding
• Abstract summary: FORGE is a model-agnostic method that induces a query-conditioned geometry on a pretrained multimodal embedding space.<n>Experiments on Video-MME and LongVideoBench at budgets of 16, 32, and 64 frames show that FORGE improves the unified selection score by 11.0-15.3 points over the strongest training-free baseline.<n>Our findings suggest that aligning the embedding space with the query's high-dimensional structure is a promising direction for inference-time video understanding.
• Score: 6.929741688224915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) have enabled long-form video understanding at a scale that was not previously possible. However, the density of relevant content decreases sharply as video sequence length increases, and exposing the model to more irrelevant content measurably reduces its accuracy. In this paper, we address the problem of maximizing query-relevant information in a frame subset selected at inference time, without training. FORGE (Frame Orthogonality in Relevance Geometry) is a model-agnostic method that induces a query-conditioned geometry on a pretrained multimodal embedding space, unifying relevance and diversity into a single objective. In this space, frames that cover independent query-relevant directions are far apart, and selecting the subset of maximum information captures diverse query-relevant content within the budget. Experiments on Video-MME and LongVideoBench at budgets of 16, 32, and 64 frames show that FORGE improves the unified keyframe selection score by 11.0-15.3 points over the strongest training-free baseline and up to doubles keyframe recall (0.415 vs. 0.204 at K=64 on Video-MME). The gains extend to question answering, where accuracy improves in every evaluated setting across eight open-source MLLMs spanning 4B to 32B parameters, by up to 8.7 points over uniform sampling and 5.2 points over the strongest baseline. Our findings suggest that aligning the embedding space with the query's high-dimensional structure is a promising direction for inference-time video understanding.

Related papers

• Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors [69.75895255395702]
  Cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation.<n>A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation.<n>Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets.
  arXiv  Detail & Related papers  (2026-07-17T07:04:31Z)
• QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding [78.36929439152566]
  We propose a Query- and Content-Aware (QCA) selection framework that can select a compact yet information-rich set of frames from long videos.<n>QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query.<n>Our method requires no additional training and can be seamlessly integrated into existing Video-LLMs.
  arXiv  Detail & Related papers  (2026-07-01T14:19:24Z)
• Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding [47.48582887762574]
  ReMem is a temporal granularity-adaptive selection framework for training-free LongVideoQA.<n>By explicitly preserving temporal information with memory mechanisms, ReMem suppresses redundancy and empowers MLLMs to perform robust multi-granular video reasoning.
  arXiv  Detail & Related papers  (2026-06-30T10:55:56Z)
• DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding [19.492762364136393]
  We present DynFrame, a framework that emits the temporal window and the sampling density as native tokens within a single autoregressive pass.<n>This learnable span-density retrieval enables acquiring multi-granularity evidence with a single retrieval step.
  arXiv  Detail & Related papers  (2026-05-26T08:16:16Z)
• Event-Anchored Frame Selection for Effective Long-Video Understanding [67.56884568828508]
  Event-Anchored Frame Selection (EFS) is a hierarchical, event-aware pipeline.<n>As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs.
  arXiv  Detail & Related papers  (2026-03-01T08:25:37Z)
• Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding [43.587729230845525]
  Current methods typically select frames with high relevance to a given query.<n>We introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework.<n>WFS-SB significantly boosts LVLM performance, improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench.
  arXiv  Detail & Related papers  (2026-02-28T07:18:07Z)
• LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoning [73.90466023069125]
  We propose LOVE-R1, a model that can adaptively zoom in on a video clip.<n>The model is first provided with densely sampled frames but in a small resolution.<n>If some spatial details are needed, the model can zoom in on a clip of interest with a large frame resolution.
  arXiv  Detail & Related papers  (2025-09-29T13:43:55Z)
• An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubes [85.00111442236499]
  This paper presents textbfQuicksviewer, an LMM with new perceiving paradigm that partitions a video of nontemporal density into varying cubes using Gumbel Softmax.<n>We train the model from a language backbone through three progressive stages, each incorporating lengthy videos on average of 420s/1fps thanks to the perceiving efficiency.<n>With only 0.8M total video-text samples for training, our model outperforms the direct baseline employing a fixed partitioning strategy by a maximum of 8.72 in accuracy.
  arXiv  Detail & Related papers  (2025-04-21T17:57:21Z)
• BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding [51.49345400300556]
  Large video-language models (VLMs) have demonstrated promising progress in various video understanding tasks.<n>Traditional approaches, such as uniform frame sampling, often inevitably allocate resources to irrelevant content.<n>We introduce BOLT, a method to BOost Large VLMs without additional Training through a comprehensive study of frame selection strategies.
  arXiv  Detail & Related papers  (2025-03-27T13:18:40Z)
• Semi-Parametric Video-Grounded Text Generation [21.506377836451577]
  In this paper, we propose a semi-parametric video-grounded text generation model, SeViT. Treating a video as an external data store, SeViT includes a non-parametric frame retriever to select a few query-relevant frames. Experimental results demonstrate our method has a significant advantage in longer videos and causal video understanding.
  arXiv  Detail & Related papers  (2023-01-27T03:00:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.