Transferable-guided Attention Is All You Need for Video Domain Adaptation

• URL: http://arxiv.org/abs/2407.01375v2
• Date: Tue, 17 Sep 2024 10:35:46 GMT
• Title: Transferable-guided Attention Is All You Need for Video Domain Adaptation
• Authors: André Sacilotti, Samuel Felipe dos Santos, Nicu Sebe, Jurandy Almeida,
• Abstract summary: Unsupervised adaptation (UDA) in videos is a challenging task that remains not well explored compared to image-based UDA techniques. Our key idea is to use transformer layers as a feature encoder and incorporate spatial and temporal transferability relationships into the attention mechanism. A Transferable-guided Attention (TransferAttn) framework is then developed to exploit the capacity of the transformer to adapt cross-domain knowledge.
• Score: 42.642008092347986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unsupervised domain adaptation (UDA) in videos is a challenging task that remains not well explored compared to image-based UDA techniques. Although vision transformers (ViT) achieve state-of-the-art performance in many computer vision tasks, their use in video UDA has been little explored. Our key idea is to use transformer layers as a feature encoder and incorporate spatial and temporal transferability relationships into the attention mechanism. A Transferable-guided Attention (TransferAttn) framework is then developed to exploit the capacity of the transformer to adapt cross-domain knowledge across different backbones. To improve the transferability of ViT, we introduce a novel and effective module, named Domain Transferable-guided Attention Block (DTAB). DTAB compels ViT to focus on the spatio-temporal transferability relationship among video frames by changing the self-attention mechanism to a transferability attention mechanism. Extensive experiments were conducted on UCF-HMDB, Kinetics-Gameplay, and Kinetics-NEC Drone datasets, with different backbones, like ResNet101, I3D, and STAM, to verify the effectiveness of TransferAttn compared with state-of-the-art approaches. Also, we demonstrate that DTAB yields performance gains when applied to other state-of-the-art transformer-based UDA methods from both video and image domains. Our code is available at https://github.com/Andre-Sacilotti/transferattn-project-code.

Related papers

• Feature Fusion Transferability Aware Transformer for Unsupervised Domain Adaptation [1.9035011984138845]
  Unsupervised domain adaptation (UDA) aims to leverage the knowledge learned from labeled source domains to improve performance on unlabeled target domains. Recent research has shown promise in applying Vision Transformers (ViTs) to this task. We propose a novel Feature Fusion Transferability Aware Transformer (FFTAT) to enhance ViT performance in UDA tasks.
  arXiv  Detail & Related papers  (2024-11-10T22:23:12Z)
• Vision Transformer-based Adversarial Domain Adaptation [5.611768906855499]
  Vision transformer (ViT) has attracted tremendous attention since its emergence and has been widely used in various computer vision tasks. In this paper, we fill this gap by employing the ViT as the feature extractor in adversarial domain adaptation. We empirically demonstrate that ViT can be a plug-and-play component in adversarial domain adaptation.
  arXiv  Detail & Related papers  (2024-04-24T11:41:28Z)
• DAT++: Spatially Dynamic Vision Transformer with Deformable Attention [87.41016963608067]
  We present Deformable Attention Transformer ( DAT++), a vision backbone efficient and effective for visual recognition. DAT++ achieves state-of-the-art results on various visual recognition benchmarks, with 85.9% ImageNet accuracy, 54.5 and 47.0 MS-COCO instance segmentation mAP, and 51.5 ADE20K semantic segmentation mIoU.
  arXiv  Detail & Related papers  (2023-09-04T08:26:47Z)
• Efficient Attention-free Video Shift Transformers [56.87581500474093]
  This paper tackles the problem of efficient video recognition. Video transformers have recently dominated the efficiency (top-1 accuracy vs FLOPs) spectrum. We extend our formulation in the video domain to construct Video Affine-Shift Transformer.
  arXiv  Detail & Related papers  (2022-08-23T17:48:29Z)
• Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective [37.45565756522847]
  We consider the generation of cross-domain videos from two sets of latent factors. TranSVAE framework is then developed to model such generation. Experiments on the UCF-HMDB, Jester, and Epic-Kitchens datasets verify the effectiveness and superiority of TranSVAE.
  arXiv  Detail & Related papers  (2022-08-15T17:59:31Z)
• Unsupervised Domain Adaptation for Video Transformers in Action Recognition [76.31442702219461]
  We propose a simple and novel UDA approach for video action recognition. Our approach builds a robust source model that better generalises to target domain. We report results on two video action benchmarks recognition for UDA.
  arXiv  Detail & Related papers  (2022-07-26T12:17:39Z)
• Can Vision Transformers Perform Convolution? [78.42076260340869]
  We prove that a single ViT layer with image patches as the input can perform any convolution operation constructively. We provide a lower bound on the number of heads for Vision Transformers to express CNNs.
  arXiv  Detail & Related papers  (2021-11-02T03:30:17Z)
• TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation [54.61786380919243]
  Unsupervised domain adaptation (UDA) aims to transfer the knowledge learnt from a labeled source domain to an unlabeled target domain. Previous work is mainly built upon convolutional neural networks (CNNs) to learn domain-invariant representations. With the recent exponential increase in applying Vision Transformer (ViT) to vision tasks, the capability of ViT in adapting cross-domain knowledge remains unexplored in the literature.
  arXiv  Detail & Related papers  (2021-08-12T22:37:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.