Frame-Voyager: Learning to Query Frames for Video Large Language Models

• URL: http://arxiv.org/abs/2410.03226v2
• Date: Mon, 7 Oct 2024 03:01:01 GMT
• Title: Frame-Voyager: Learning to Query Frames for Video Large Language Models
• Authors: Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xiaolei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, Qianru Sun,
• Abstract summary: Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos. We propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task.
• Score: 33.84793162102087
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.

Related papers

• TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models [52.590072198551944]
  Recent advances in multimodal Large Language Models (LLMs) have shown great success in understanding multi-modal contents. For video understanding tasks, training-based video LLMs are difficult to build due to the scarcity of high-quality, curated video-text paired data. In this work, we explore the limitations of the existing compression strategies for building a training-free video LLM.
  arXiv  Detail & Related papers  (2024-11-17T13:08:29Z)
• VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding [63.075626670943116]
  We introduce a cutting-edge framework, VaQuitA, designed to refine the synergy between video and textual information. At the data level, instead of sampling frames uniformly, we implement a sampling method guided by CLIP-score rankings. At the feature level, we integrate a trainable Video Perceiver alongside a Visual-Query Transformer.
  arXiv  Detail & Related papers  (2023-12-04T19:48:02Z)
• An Empirical Study of Frame Selection for Text-to-Video Retrieval [62.28080029331507]
  Text-to-video retrieval (TVR) aims to find the most relevant video in a large video gallery given a query text. Existing methods typically select a subset of frames within a video to represent the video content for TVR. In this paper, we make the first empirical study of frame selection for TVR.
  arXiv  Detail & Related papers  (2023-11-01T05:03:48Z)
• FrameRS: A Video Frame Compression Model Composed by Self supervised Video Frame Reconstructor and Key Frame Selector [9.896415488558036]
  We present frame reconstruction model: FrameRS. It consists self-supervised video frame reconstructor and key frame selector. The frame reconstructor, FrameMAE, is developed by adapting the principles of the Masked Autoencoder for Images (MAE) for video context. The key frame selector, Frame Selector, is built on CNN architecture.
  arXiv  Detail & Related papers  (2023-09-16T19:30:05Z)
• Search-Map-Search: A Frame Selection Paradigm for Action Recognition [21.395733318164393]
  Frame selection aims to extract the most informative and representative frames to help a model better understand video content. Existing frame selection methods either individually sample frames based on per-frame importance prediction, or adopt reinforcement learning agents to find representative frames in succession. We propose a Search-Map-Search learning paradigm which combines the advantages of search and supervised learning to select the best combination of frames from a video as one entity.
  arXiv  Detail & Related papers  (2023-04-20T13:49:53Z)
• Efficient Cross-Modal Video Retrieval with Meta-Optimized Frames [39.03408879727955]
  Cross-modal video retrieval aims to retrieve semantically relevant videos given a text as a query. A common and simple solution is to uniformly sample a small number of frames from the video as input to ViT. To get free from this trade-off, this paper introduces an automatic video compression method based on a bilevel optimization program.
  arXiv  Detail & Related papers  (2022-10-16T05:35:00Z)
• Contrastive Video-Language Learning with Fine-grained Frame Sampling [54.542962813921214]
  FineCo is an approach to better learn video and language representations with a fine-grained contrastive objective operating on video frames. It helps distil a video by selecting the frames that are semantically equivalent to the text, improving cross-modal correspondence.
  arXiv  Detail & Related papers  (2022-10-10T22:48:08Z)
• Multimodal Frame-Scoring Transformer for Video Summarization [4.266320191208304]
  Multimodal Frame-Scoring Transformer (MFST) framework exploiting visual, text and audio features and scoring a video with respect to frames. MFST framework first extracts each modality features (visual-text-audio) using pretrained encoders. MFST trains the multimodal frame-scoring transformer that uses video-text-audio representations as inputs and predicts frame-level scores.
  arXiv  Detail & Related papers  (2022-07-05T05:14:15Z)
• Optimizing Video Prediction via Video Frame Interpolation [53.16726447796844]
  We present a new optimization framework for video prediction via video frame, inspired by photo-realistic results of video framescapes. Our framework is based on optimization with a pretrained differentiable video frame module without the need for a training dataset. Our approach outperforms other video prediction methods that require a large amount of training data or extra semantic information.
  arXiv  Detail & Related papers  (2022-06-27T17:03:46Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.