An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling

• URL: http://arxiv.org/abs/2209.01540v5
• Date: Wed, 31 May 2023 20:55:08 GMT
• Title: An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
• Authors: Tsu-Jui Fu and Linjie Li and Zhe Gan and Kevin Lin and William Yang Wang and Lijuan Wang and Zicheng Liu
• Abstract summary: Masked visual modeling (MVM) has been recently proven effective for visual pre-training. We systematically examine the potential of MVM in the context of VidL learning. We show VIOLETv2 pre-trained with MVM achieves notable improvements on 13 VidL benchmarks.
• Score: 152.75131627307567
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked visual modeling (MVM) has been recently proven effective for visual pre-training. While similar reconstructive objectives on video inputs (e.g., masked frame modeling) have been explored in video-language (VidL) pre-training, previous studies fail to find a truly effective MVM strategy that can largely benefit the downstream performance. In this work, we systematically examine the potential of MVM in the context of VidL learning. Specifically, we base our study on a fully end-to-end VIdeO-LanguagE Transformer (VIOLET), where the supervision from MVM training can be backpropagated to the video pixel space. In total, eight different reconstructive targets of MVM are explored, from low-level pixel values and oriented gradients to high-level depth maps, optical flow, discrete visual tokens, and latent visual features. We conduct comprehensive experiments and provide insights into the factors leading to effective MVM training, resulting in an enhanced model VIOLETv2. Empirically, we show VIOLETv2 pre-trained with MVM objective achieves notable improvements on 13 VidL benchmarks, ranging from video question answering, video captioning, to text-to-video retrieval.

Related papers

• The 1st Solution for 4th PVUW MeViS Challenge: Unleashing the Potential of Large Multimodal Models for Referring Video Segmentation [31.44879457190659]
  We propose a simple and effective inference optimization method to fully unleash the potential of LMMs in referring video segmentation. Our solution achieved 61.98% J&F on the MeViS test set and ranked 1st place in the 4th PVUW Challenge MeViS Track at CVPR 2025.
  arXiv  Detail & Related papers  (2025-04-07T15:24:54Z)
• 4th PVUW MeViS 3rd Place Report: Sa2VA [105.88675577642204]
  We show that with a simple modification to the test time inference method on stronger MLLMs, we can lead to stronger results on MeVIS. In particular, we adopt the recent method Sa2VA, a unified model for dense grounded understanding of both images and videos.
  arXiv  Detail & Related papers  (2025-04-01T07:06:47Z)
• Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension [95.63899307791665]
  Vision Value Model (VisVM) can guide VLM inference-time search to generate responses with better visual comprehension. In this paper, we present VisVM that can guide VLM inference-time search to generate responses with better visual comprehension.
  arXiv  Detail & Related papers  (2024-12-04T20:35:07Z)
• How Well Can Vision Language Models See Image Details? [53.036922527685064]
  We introduce a pixel value prediction task to explore "How Well Can Vision Language Models See Image Details?" Our research reveals that incorporating pixel value prediction as one of the VLM pre-training tasks and vision encoder adaptation markedly boosts VLM performance on downstream image-language understanding tasks.
  arXiv  Detail & Related papers  (2024-08-07T17:59:40Z)
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs [56.391404083287235]
  We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes.
  arXiv  Detail & Related papers  (2024-06-24T17:59:42Z)
• Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models [29.825619120260164]
  This paper addresses the challenge by leveraging the visual commonalities between images and videos to evolve image-LVLMs into video-LVLMs. We present a cost-effective video-LVLM that enhances model architecture, introduces innovative training strategies, and identifies the most effective types of video instruction data.
  arXiv  Detail & Related papers  (2024-06-12T09:22:45Z)
• E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizer [5.7254320553764]
  E-ViLM is able to learn expressive representations from Video-Language corpus and generalize well to extensive Video-Language tasks. Our model reaches $39.3$% Top-$1$ accuracy on the MSRVTT benchmark, retaining $91.4$% of the accuracy of state-of-the-art larger VL architecture.
  arXiv  Detail & Related papers  (2023-11-28T22:57:17Z)
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection [27.04277811443469]
  Video-LLaVA learns from a mixed dataset of images and videos, mutually enhancing each other. Video-LLaVA superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4 image benchmark toolkits.
  arXiv  Detail & Related papers  (2023-11-16T10:59:44Z)
• Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning [123.63301596019522]
  Masked video distillation (MVD) is a simple yet effective two-stage masked feature modeling framework for video representation learning. For the choice of teacher models, we observe that students taught by video teachers perform better on temporally-heavy video tasks. We design a spatial-temporal co-teaching method for MVD to leverage the advantage of different teachers.
  arXiv  Detail & Related papers  (2022-12-08T18:59:59Z)
• Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models [67.31684040281465]
  We present textbfMOV, a simple yet effective method for textbfMultimodal textbfOpen-textbfVocabulary video classification. In MOV, we directly use the vision encoder from pre-trained VLMs with minimal modifications to encode video, optical flow and audio spectrogram.
  arXiv  Detail & Related papers  (2022-07-15T17:59:11Z)
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling [88.30109041658618]
  A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. We present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs.
  arXiv  Detail & Related papers  (2021-11-24T18:31:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.