M-LLM Based Video Frame Selection for Efficient Video Understanding

• URL: http://arxiv.org/abs/2502.19680v2
• Date: Wed, 26 Mar 2025 21:14:41 GMT
• Title: M-LLM Based Video Frame Selection for Efficient Video Understanding
• Authors: Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, Trishul Chilimbi,
• Abstract summary: We propose a light-weight M-LLM-based frame selection method that adaptively select frames that are more relevant to users' queries.<n>The selected frames are then digested by a frozen downstream video M-LLM for visual reasoning and question answering.
• Score: 60.93714759178143
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in Multi-Modal Large Language Models (M-LLMs) show promising results in video reasoning. Popular Multi-Modal Large Language Model (M-LLM) frameworks usually apply naive uniform sampling to reduce the number of video frames that are fed into an M-LLM, particularly for long context videos. However, it could lose crucial context in certain periods of a video, so that the downstream M-LLM may not have sufficient visual information to answer a question. To attack this pain point, we propose a light-weight M-LLM -based frame selection method that adaptively select frames that are more relevant to users' queries. In order to train the proposed frame selector, we introduce two supervision signals (i) Spatial signal, where single frame importance score by prompting a M-LLM; (ii) Temporal signal, in which multiple frames selection by prompting Large Language Model (LLM) using the captions of all frame candidates. The selected frames are then digested by a frozen downstream video M-LLM for visual reasoning and question answering. Empirical results show that the proposed M-LLM video frame selector improves the performances various downstream video Large Language Model (video-LLM) across medium (ActivityNet, NExT-QA) and long (EgoSchema, LongVideoBench) context video question answering benchmarks.

Related papers

• FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understanding [70.56829394569938]
  We propose Frame Selection Augmented Generation (FRAG) to process long inputs without long context LMMs. The core of the selection process is done by scoring each frame independently, which does not require long context processing. We show that FRAG consistently improves the performance and achieves state-of-the-art performances for both long video and long document understanding.
  arXiv  Detail & Related papers  (2025-04-24T11:19:18Z)
• Adaptive Keyframe Sampling for Long Video Understanding [75.7837692594814]
  This paper presents a simple yet effective algorithm named Adaptive Keyframe Sampling (AKS) It inserts a plug-and-play module known as Adaptive Keyframe Sampling (AKS) which aims to maximize the useful information with a fixed number of video tokens. Experiments on two long video understanding benchmarks validate that AKS improves video QA accuracy upon selecting informative encounters.
  arXiv  Detail & Related papers  (2025-02-28T17:46:29Z)
• MAMS: Model-Agnostic Module Selection Framework for Video Captioning [11.442879458679144]
  Existing multi-modal video captioning methods typically extract a fixed number of frames, which raises critical challenges.<n>This paper proposes the first model-agnostic module selection framework in video captioning.<n>Our experiments on three different benchmark datasets demonstrate that the proposed framework significantly improves the performance of three recent video captioning models.
  arXiv  Detail & Related papers  (2025-01-30T11:10:18Z)
• VidCtx: Context-aware Video Question Answering with Image Models [15.1350316858766]
  We introduce VidCtx, a novel training-free VideoQA framework which integrates both visual information from input frames and textual descriptions of others frames.<n>Experiments show that VidCtx achieves competitive performance among approaches that rely on open models.
  arXiv  Detail & Related papers  (2024-12-23T09:26:38Z)
• Frame-Voyager: Learning to Query Frames for Video Large Language Models [33.84793162102087]
  Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos. We propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task.
  arXiv  Detail & Related papers  (2024-10-04T08:26:06Z)
• Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner [53.671484175063995]
  Video-LLMs are pre-trained to process short videos, limiting their broader application for understanding longer video content. We introduce an alternative video token rearrangement technique that circumvents limitations imposed by the fixed video encoder and alignment projector.
  arXiv  Detail & Related papers  (2024-09-19T17:59:55Z)
• Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA [40.54207548074378]
  Long-form videos that span across wide temporal intervals are highly information redundant.<n>All information necessary to generate a correct response can often be contained within a small subset of frames.
  arXiv  Detail & Related papers  (2024-06-13T17:59:16Z)
• MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding [66.56100008577134]
  This study focuses on designing an efficient and effective model for long-term video understanding. We propose to process videos in an online manner and store past video information in a memory bank. Our model can achieve state-of-the-art performances across multiple datasets.
  arXiv  Detail & Related papers  (2024-04-08T17:59:24Z)
• Contrastive Video-Language Learning with Fine-grained Frame Sampling [54.542962813921214]
  FineCo is an approach to better learn video and language representations with a fine-grained contrastive objective operating on video frames. It helps distil a video by selecting the frames that are semantically equivalent to the text, improving cross-modal correspondence.
  arXiv  Detail & Related papers  (2022-10-10T22:48:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.