DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks

• URL: http://arxiv.org/abs/2304.00571v2
• Date: Fri, 7 Apr 2023 02:55:28 GMT
• Title: DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks
• Authors: Qiangqiang Wu and Tianyu Yang and Ziquan Liu and Baoyuan Wu and Ying Shan and Antoni B. Chan
• Abstract summary: Masked autoencoder (MAE) pretraining on videos for matching-based downstream tasks, including visual object tracking (VOT) and video object segmentation (VOS) We propose DropMAE, which adaptively performs spatial-attention dropout in the frame reconstruction to facilitate temporal correspondence learning in videos. Our model sets new state-of-the-art performance on 8 out of 9 highly competitive video tracking and segmentation datasets.
• Score: 76.24996889649744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we study masked autoencoder (MAE) pretraining on videos for matching-based downstream tasks, including visual object tracking (VOT) and video object segmentation (VOS). A simple extension of MAE is to randomly mask out frame patches in videos and reconstruct the frame pixels. However, we find that this simple baseline heavily relies on spatial cues while ignoring temporal relations for frame reconstruction, thus leading to sub-optimal temporal matching representations for VOT and VOS. To alleviate this problem, we propose DropMAE, which adaptively performs spatial-attention dropout in the frame reconstruction to facilitate temporal correspondence learning in videos. We show that our DropMAE is a strong and efficient temporal matching learner, which achieves better finetuning results on matching-based tasks than the ImageNetbased MAE with 2X faster pre-training speed. Moreover, we also find that motion diversity in pre-training videos is more important than scene diversity for improving the performance on VOT and VOS. Our pre-trained DropMAE model can be directly loaded in existing ViT-based trackers for fine-tuning without further modifications. Notably, DropMAE sets new state-of-the-art performance on 8 out of 9 highly competitive video tracking and segmentation datasets. Our code and pre-trained models are available at https://github.com/jimmy-dq/DropMAE.git.

Related papers

• Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders [89.12558126877532]
  We propose CropMAE, an alternative approach to the Siamese pre-training introduced by SiamMAE. Our method exclusively considers pairs of cropped images sourced from the same image but cropped differently, deviating from the conventional pairs of frames extracted from a video. CropMAE achieves the highest masking ratio to date (98.5%), enabling the reconstruction of images using only two visible patches.
  arXiv  Detail & Related papers  (2024-03-26T16:04:19Z)
• Multi-entity Video Transformers for Fine-Grained Video Representation Learning [36.31020249963468]
  We re-examine the design of transformer architectures for video representation learning. A salient aspect of our self-supervised method is the improved integration of spatial information in the temporal pipeline. Our Multi-entity Video Transformer (MV-Former) architecture achieves state-of-the-art results on multiple fine-grained video benchmarks.
  arXiv  Detail & Related papers  (2023-11-17T21:23:12Z)
• Concatenated Masked Autoencoders as Spatial-Temporal Learner [6.475592804311682]
  We introduce the Concatenated Masked Autoencoders (CatMAE) as a spatial-temporal learner for self-supervised video representation learning. We propose a new data augmentation strategy, Video-Reverse (ViRe), which uses reversed video frames as the model's reconstruction targets.
  arXiv  Detail & Related papers  (2023-11-02T03:08:26Z)
• Unmasked Teacher: Towards Training-Efficient Video Foundation Models [50.19560876891811]
  Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. Our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding.
  arXiv  Detail & Related papers  (2023-03-28T15:39:28Z)
• You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos [56.676761067861236]
  Given an untrimmed video, temporal sentence grounding aims to locate a target moment semantically according to a sentence query. Previous respectable works have made decent success, but they only focus on high-level visual features extracted from decoded frames. We propose a new setting, compressed-domain TSG, which directly utilizes compressed videos rather than fully-decompressed frames as the visual input.
  arXiv  Detail & Related papers  (2023-03-14T12:53:27Z)
• Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoders [46.38458873424361]
  Masked autoencoders (MAEs) have emerged recently as art self-supervised representation learners. In this work we present a motion-aware variant -- MotionMAE. Our model is designed to additionally predict the corresponding motion structure information over time.
  arXiv  Detail & Related papers  (2022-10-09T03:22:15Z)
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling [88.30109041658618]
  A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. We present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs.
  arXiv  Detail & Related papers  (2021-11-24T18:31:20Z)
• Attention-guided Temporal Coherent Video Object Matting [78.82835351423383]
  We propose a novel deep learning-based object matting method that can achieve temporally coherent matting results. Its key component is an attention-based temporal aggregation module that maximizes image matting networks' strength. We show how to effectively solve the trimap generation problem by fine-tuning a state-of-the-art video object segmentation network.
  arXiv  Detail & Related papers  (2021-05-24T17:34:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.