Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs

• URL: http://arxiv.org/abs/2608.05592v1
• Date: Thu, 06 Aug 2026 04:27:59 GMT
• Title: Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs
• Abstract summary: Video(VR) organizes each video into a question-agnostic temporal hierarchy.<n> nodes capture broad storyline context and event progression.<n>Lower-level nodes preserve fine-grained local details and evidence-bearing moments.
• Score: 16.29545034560019
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) have achieved strong progress in video understanding, yet it remains challenging because the token limitation makes MLLMs difficult to capture temporally sparse evidence. Existing methods typically rely on uniform sampling, or frame selection, but these strategies usually optimize either broad temporal coverage or local relevance, making it difficult to preserve both global storyline context and fine-grained evidence. We propose VideoRouter(VR) that rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. It first organizes each video into a question-agnostic temporal hierarchy, which partition the video into coarse-to-fine temporally coherent segments. In this hierarchy, upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This naturally gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router to determine which view is better supported by the selected evidence and select the final answer. We validate the effectiveness of the proposed design through extensive experiments, showing that the verification-guided router effectively coordinates global and local reasoning, and that, on VideoMME, our method outperforms state-of-the-art frame selection methods by 2.9 points, respectively, under the LLaVA-Video-7B backbone. We will release the code.

Related papers

• VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding [61.00809248898572]
  Long video understanding is a problem in which an agent reads the video turn by turn along the temporal axis, deciding at each turn how fast to watch, what evidence to retain, when to revisit uncertain segments, and when to stop and answer.<n>We propose textbfVideoScout, a multi-turn reasoning agent that instantiates the SEA paradigm through adaptive reasoning pacing.<n>Experiments on long video understanding and reasoning benchmarks demonstrate that our 7B model achieves strong performance compared with existing trained 7B agentic models.
  arXiv  Detail & Related papers  (2026-09-14T14:04:03Z)
• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding [55.75126327155788]
  RIDGE is a frame selection framework that reads the frame-query similarity curve as a temporal signal.<n>It partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget.<n>It achieves the best performance in most settings and remains competitive in the others.
  arXiv  Detail & Related papers  (2026-08-30T18:40:51Z)
• VisualRouter: Query-Grounded Visual Sampling for Long Video Understanding [60.161767365040554]
  We propose a training-free and plug-and-play framework for query-grounded visual sampling.<n>Visual first classifies each query as either global or local and then applies the corresponding sampling strategy.<n>Experiments show that Visual consistently improves multiple LVLMs over uniform sampling, achieving gains of 5.2%, 7.7%, and 11.6% on Video-MME, LongVideoBench, and MLVU with Qwen2.5-VL-7B, and outperforming existing training-free visual sampling methods under the same setting.
  arXiv  Detail & Related papers  (2026-07-30T16:23:10Z)
• HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization [16.58089288863005]
  We propose HAS, a Highlight-guided Attention Steering method for video summarization.<n>We evaluate HAS on a variety of benchmarks, and it has shown convincing performance in video summarization.
  arXiv  Detail & Related papers  (2026-07-20T14:27:06Z)
• VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding [40.699624658181456]
  VideoDetective is a framework that integrates query-to-segment relevance and inter-segment affinity for effective clue hunting in long-video question answering.<n>Our method consistently achieves substantial gains across a wide range of mainstream MLLMs on representative benchmarks, with accuracy improvements of up to 7.5% on VideoMME-long.
  arXiv  Detail & Related papers  (2026-03-23T17:59:51Z)
• SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM [36.28285195488772]
  Large language models (LLMs) have demonstrated exceptional capabilities in text understanding.<n>Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information.<n>This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding.
  arXiv  Detail & Related papers  (2026-02-03T14:39:16Z)
• SeViCES: Unifying Semantic-Visual Evidence Consensus for Long Video Understanding [36.30263540665245]
  We propose a framework for effective and reliable long video understanding.<n>SeViCES is training-free and model-agnostic, and introduces two key components.<n> experiments on long video understanding benchmarks show that SeViCES consistently outperforms state-of-the-art methods in both accuracy and robustness.
  arXiv  Detail & Related papers  (2025-10-23T14:55:28Z)
• MHSCNet: A Multimodal Hierarchical Shot-aware Convolutional Network for Video Summarization [61.69587867308656]
  We propose a multimodal hierarchical shot-aware convolutional network, denoted as MHSCNet, to enhance the frame-wise representation. Based on the learned shot-aware representations, MHSCNet can predict the frame-level importance score in the local and global view of the video.
  arXiv  Detail & Related papers  (2022-04-18T14:53:33Z)
• Local-Global Associative Frame Assemble in Video Re-ID [57.7470971197962]
  Noisy and unrepresentative frames in automatically generated object bounding boxes from video sequences cause challenges in learning discriminative representations in video re-identification (Re-ID) Most existing methods tackle this problem by assessing the importance of video frames according to either their local part alignments or global appearance correlations separately. In this work, we explore jointly both local alignments and global correlations with further consideration of their mutual promotion/reinforcement.
  arXiv  Detail & Related papers  (2021-10-22T19:07:39Z)
• Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation [96.67525775629444]
  Action segmentation refers to inferring boundaries of semantically consistent visual concepts in videos. We present a fully automatic and unsupervised approach for segmenting actions in a video that does not require any training. Our proposal is an effective temporally-weighted hierarchical clustering algorithm that can group semantically consistent frames of the video.
  arXiv  Detail & Related papers  (2021-03-20T23:30:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.