VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding

• URL: http://arxiv.org/abs/2501.13106v3
• Date: Tue, 28 Jan 2025 11:05:18 GMT
• Title: VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding
• Authors: Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, Deli Zhao,
• Abstract summary: VideoLLaMA3 is a more advanced multimodal foundation model for image and video understanding. VideoLLaMA3 has four training stages: Vision Adaptation, Vision-Language Alignment, Fine-tuning, and Video-centric Fine-tuning. VideoLLaMA3 achieves compelling performances in both image and video understanding benchmarks.
• Score: 59.020450264301026
• License:
• Abstract: In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation model for image and video understanding. The core design philosophy of VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the vision-centric training paradigm and vision-centric framework design. The key insight of our vision-centric training paradigm is that high-quality image-text data is crucial for both image and video understanding. Instead of preparing massive video-text datasets, we focus on constructing large-scale and high-quality image-text datasets. VideoLLaMA3 has four training stages: 1) Vision Encoder Adaptation, which enables vision encoder to accept images of variable resolutions as input; 2) Vision-Language Alignment, which jointly tunes the vision encoder, projector, and LLM with large-scale image-text data covering multiple types (including scene images, documents, charts) as well as text-only data. 3) Multi-task Fine-tuning, which incorporates image-text SFT data for downstream tasks and video-text data to establish a foundation for video understanding. 4) Video-centric Fine-tuning, which further improves the model's capability in video understanding. As for the framework design, to better capture fine-grained details in images, the pretrained vision encoder is adapted to encode images of varying sizes into vision tokens with corresponding numbers, rather than a fixed number of tokens. For video inputs, we reduce the number of vision tokens according to their similarity so that the representation of videos will be more precise and compact. Benefit from vision-centric designs, VideoLLaMA3 achieves compelling performances in both image and video understanding benchmarks.

Related papers

• Unifying Specialized Visual Encoders for Video Language Models [38.89164471290503]
  Video Large Language Models (VideoLLMs) rely on a single vision encoder for all of their visual processing. Our method, MERV, Multi-Encoder Representation of Videos, instead leverages multiple frozen visual encoders to create a unified representation of a video. MERV is up to 3.7% better in accuracy than Video-LLaVA across the standard suite video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-01-02T18:59:45Z)
• Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding [41.59673370285659]
  We present a comprehensive study that probes various visual encoding models for 3D scene understanding. Our evaluation spans seven vision foundation encoders, including image-based, video-based, and 3D foundation models. Our evaluations yield key findings: DINOv2 demonstrates superior performance, video models excel in object-level tasks, geometric diffusion models benefit tasks, and language-pretrained models show unexpected limitations in language-related tasks.
  arXiv  Detail & Related papers  (2024-09-05T17:59:56Z)
• How Well Can Vision Language Models See Image Details? [53.036922527685064]
  We introduce a pixel value prediction task to explore "How Well Can Vision Language Models See Image Details?" Our research reveals that incorporating pixel value prediction as one of the VLM pre-training tasks and vision encoder adaptation markedly boosts VLM performance on downstream image-language understanding tasks.
  arXiv  Detail & Related papers  (2024-08-07T17:59:40Z)
• OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding [112.87441334765693]
  OMG-LLaVA is a new framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model.
  arXiv  Detail & Related papers  (2024-06-27T17:59:01Z)
• VisionGPT-3D: A Generalized Multimodal Agent for Enhanced 3D Vision Understanding [47.58359136198136]
  VisionGPT-3D provides a versatile multimodal framework building upon the strengths of multimodal foundation models. It seamlessly integrates various SOTA vision models and brings the automation in the selection of SOTA vision models. It identifies the suitable 3D mesh creation algorithms corresponding to 2D depth maps analysis, generates optimal results based on diverse multimodal inputs.
  arXiv  Detail & Related papers  (2024-03-14T16:13:00Z)
• NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos [51.409547544747284]
  NPF-200 is the first large-scale multi-modal dataset of purely non-photorealistic videos with eye fixations. We conduct a series of analyses to gain deeper insights into this task. We propose a universal frequency-aware multi-modal non-photorealistic saliency detection model called NPSNet.
  arXiv  Detail & Related papers  (2023-08-23T14:25:22Z)
• Let's Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought [62.619076257298204]
  We motivate framing video reasoning as the sequential understanding of a small number of video reasonings. We introduce VIP, an inference-time challenge dataset designed to explore models' reasoning capabilities through video chain-of-thought. We benchmark GPT-4, GPT-3, and VICUNA on VIP, demonstrate the performance gap in complex video reasoning tasks, and encourage future work.
  arXiv  Detail & Related papers  (2023-05-23T10:26:42Z)
• VLMAE: Vision-Language Masked Autoencoder [21.97700040013084]
  We propose a vision-language masked autoencoder framework (VLMAE) for vision-language pre-training. VLMAE employs visual generative learning, facilitating the model to acquire fine-grained and unbiased features.
  arXiv  Detail & Related papers  (2022-08-19T14:39:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.