Self-supervised Video Transformer

• URL: http://arxiv.org/abs/2112.01514v1
• Date: Thu, 2 Dec 2021 18:59:02 GMT
• Title: Self-supervised Video Transformer
• Authors: Kanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, Michael Ryoo
• Abstract summary: From a given video, we create local and global views with varying spatial sizes and frame rates. Our self-supervised objective seeks to match the features of different views representing the same video to be intemporal. Our approach performs well on four action benchmarks and converges faster with small batch sizes.
• Score: 46.295395772938214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose self-supervised training for video transformers using unlabelled video data. From a given video, we create local and global spatiotemporal views with varying spatial sizes and frame rates. Our self-supervised objective seeks to match the features of these different views representing the same video, to be invariant to spatiotemporal variations in actions. To the best of our knowledge, the proposed approach is the first to alleviate the dependency on negative samples or dedicated memory banks in Self-supervised Video Transformer (SVT). Further, owing to the flexibility of Transformer models, SVT supports slow-fast video processing within a single architecture using dynamically adjusted positional encodings and supports long-term relationship modeling along spatiotemporal dimensions. Our approach performs well on four action recognition benchmarks (Kinetics-400, UCF-101, HMDB-51, and SSv2) and converges faster with small batch sizes. Code: https://git.io/J1juJ

Related papers

• A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames [54.90226700939778]
  We build on the common paradigm of transferring large-scale, image--text models to video via shallow temporal fusion. We expose two limitations to the approach: (1) decreased spatial capabilities, likely due to poor video--language alignment in standard video datasets, and (2) higher memory consumption, bottlenecking the number of frames that can be processed.
  arXiv  Detail & Related papers  (2023-12-12T16:10:19Z)
• Dual-path Adaptation from Image to Video Transformers [62.056751480114784]
  We efficiently transfer the surpassing representation power of the vision foundation models, such as ViT and Swin, for video understanding with only a few trainable parameters. We propose a novel DualPath adaptation separated into spatial and temporal adaptation paths, where a lightweight bottleneck adapter is employed in each transformer block.
  arXiv  Detail & Related papers  (2023-03-17T09:37:07Z)
• SVFormer: Semi-supervised Video Transformer for Action Recognition [88.52042032347173]
  We introduce SVFormer, which adopts a steady pseudo-labeling framework to cope with unlabeled video samples. In addition, we propose a temporal warping to cover the complex temporal variation in videos. In particular, SVFormer outperforms the state-of-the-art by 31.5% with fewer training epochs under the 1% labeling rate of Kinetics-400.
  arXiv  Detail & Related papers  (2022-11-23T18:58:42Z)
• VDTR: Video Deblurring with Transformer [24.20183395758706]
  Videoblurring is still an unsolved problem due to the challenging-temporal modeling process. This paper presents VDTR, an effective Transformer-based model that makes the first attempt to adapt for Transformer video dering.
  arXiv  Detail & Related papers  (2022-04-17T14:22:14Z)
• Deformable Video Transformer [44.71254375663616]
  We introduce the Deformable Video Transformer (DVT), which predicts a small subset of video patches to attend for each query location based on motion information. Our model achieves higher accuracy at the same or lower computational cost, and it attains state-of-the-art results on four datasets.
  arXiv  Detail & Related papers  (2022-03-31T04:52:27Z)
• Multiview Transformers for Video Recognition [69.50552269271526]
  We present Multiview Video Recognition (MTV) for different resolutions. MTV consistently performs better than single-view counterparts in terms of accuracy and computational cost. We achieve state-of-the-art results on five standard datasets, and improve even further with large-scale pretraining.
  arXiv  Detail & Related papers  (2022-01-12T03:33:57Z)
• VideoLightFormer: Lightweight Action Recognition using Transformers [8.871042314510788]
  We propose a novel, lightweight action recognition architecture, VideoLightFormer. In a factorized fashion, we carefully extend the 2D convolutional Temporal Network with transformers. We evaluate VideoLightFormer in a high-efficiency setting on the temporally-demanding EPIC-KITCHENS-100 and Something-SV-V-Something2 datasets.
  arXiv  Detail & Related papers  (2021-07-01T13:55:52Z)
• Long-Short Temporal Contrastive Learning of Video Transformers [62.71874976426988]
  Self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results on par or better than those obtained with supervised pretraining on large-scale image datasets. Our approach, named Long-Short Temporal Contrastive Learning, enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
  arXiv  Detail & Related papers  (2021-06-17T02:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.