Towards Effective and Efficient Long Video Understanding of Multimodal Large Language Models via One-shot Clip Retrieval

• URL: http://arxiv.org/abs/2512.08410v1
• Date: Tue, 09 Dec 2025 09:40:20 GMT
• Title: Towards Effective and Efficient Long Video Understanding of Multimodal Large Language Models via One-shot Clip Retrieval
• Authors: Tao Chen, Shaobo Ju, Qiong Wu, Chenxin Fang, Kun Zhang, Jun Peng, Hui Li, Yiyi Zhou, Rongrong Ji,
• Abstract summary: We propose One-shot video-Clip based Retrieval AuGmentation (OneClip-RAG)<n>OneClip-RAG makes full use of the merits of video clips for augmented video understanding.<n>It is also equipped with a novel query-guided video chunking algorithm.
• Score: 57.88666884515147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to excessive memory overhead, most Multimodal Large Language Models (MLLMs) can only process videos of limited frames. In this paper, we propose an effective and efficient paradigm to remedy this shortcoming, termed One-shot video-Clip based Retrieval AuGmentation (OneClip-RAG). Compared with existing video RAG methods, OneClip-RAG makes full use of the merits of video clips for augmented video understanding in terms of both knowledge integrity and semantic coherence. Besides, it is also equipped with a novel query-guided video chunking algorithm that can unify clip chunking and cross-modal retrieval in one processing step, avoiding redundant computations. To improve instruction following, we further propose a new dataset called SynLongVideo and design a progressive training regime for OneClip-RAG. OneClip-RAG is plugged into five recent MLLMs and validated on a set of long-video benchmarks. Experimental results not only show the obvious performance gains by OneClip-RAG over MLLMs, e.g., boosting InternLV2 8B and Qwen2-VL 7B to the level of GPT-4o on MLVU, but also show its superior efficiency in handling long videos. e.g., enabling LLaVA-Video understand up to an hour of videos in less than 2.2 minutes on a single 4090 GPU.

Related papers

• LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding [106.23494088118571]
  LongVideo-R1 is a multimodal large language model (MLLM) agent for efficient video context navigation.<n>It infers the most informative video clip for subsequent processing.<n>The LongVideo-R1 agent is fine-tuned upon the Qwen-3-8B model through a two-stage paradigm.
  arXiv  Detail & Related papers  (2026-02-24T13:49:47Z)
• Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding [18.742640666790827]
  Think-Clip-Sample (TCS) is a training-free framework that enhances long video understanding through two key components.<n>Extensive experiments on MLVU, LongVideoBench, and VideoMME demonstrate that TCS consistently improves performance across different MLLMs.
  arXiv  Detail & Related papers  (2026-01-16T15:14:04Z)
• InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling [56.130911402831906]
  This paper aims to improve the performance of video large language models (LM) via long and rich context (LRC) modeling.<n>We develop a new version of InternVideo2.5 with focus on enhancing the original MLLMs' ability to perceive fine-grained details in videos.<n> Experimental results demonstrate this unique designML LRC greatly improves the results of video MLLM in mainstream understanding benchmarks.
  arXiv  Detail & Related papers  (2025-01-21T18:59:00Z)
• VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling [43.485687038460895]
  Long-context video modeling is critical for multimodal large language models (MLLMs)<n>This paper aims to address this issue from aspects of model architecture, training data, training strategy and evaluation benchmark.<n>We build a powerful video MLLM named VideoChat-Flash, which shows a leading performance on both mainstream long and short video benchmarks.
  arXiv  Detail & Related papers  (2024-12-31T18:01:23Z)
• Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension [83.00346826110041]
  Video-RAG is a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment.<n>Our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.
  arXiv  Detail & Related papers  (2024-11-20T07:44:34Z)
• AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction [10.579335027350263]
  AdaCM$2$ is an adaptive cross-modality memory reduction approach to video-text alignment on video streams.<n>It achieves a 4.5% improvement across multiple tasks in the LVU dataset with a GPU memory consumption reduction of up to 65%.
  arXiv  Detail & Related papers  (2024-11-19T18:04:13Z)
• VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges [39.666361965650836]
  VideoLLaMB is a framework for long video understanding.<n> SceneTiling algorithm segments videos into coherent semantic units.<n>VideoLLaMB processes up to 320 frames using a single Nvidia A100 GPU.
  arXiv  Detail & Related papers  (2024-09-02T08:52:58Z)
• A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames [57.758863967770594]
  We build on the common paradigm of transferring large-scale, image--text models to video via shallow temporal fusion.<n>We expose two limitations to the approach: (1) decreased spatial capabilities, likely due to poor video--language alignment in standard video datasets, and (2) higher memory consumption, bottlenecking the number of frames that can be processed.
  arXiv  Detail & Related papers  (2023-12-12T16:10:19Z)
• Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories [56.91664227337115]
  We introduce a collaborative memory mechanism that encodes information across multiple sampled clips of a video at each training iteration. This enables the learning of long-range dependencies beyond a single clip. Our proposed framework is end-to-end trainable and significantly improves the accuracy of video classification at a negligible computational overhead.
  arXiv  Detail & Related papers  (2021-04-02T18:59:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.