Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition

• URL: http://arxiv.org/abs/2402.12706v2
• Date: Tue, 4 Jun 2024 18:08:31 GMT
• Title: Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition
• Authors: Yuke Li, Guangyi Chen, Ben Abramowitz, Stefano Anzellott, Donglai Wei,
• Abstract summary: Few-shot action recognition aims at quickly adapting a pre-trained model to novel data. Key challenges include how to identify and leverage the transferable knowledge learned by the pre-trained model. We propose CDTD, or Causal Domain-Invariant Temporal Dynamics for knowledge transfer.
• Score: 12.522600594024112
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot action recognition aims at quickly adapting a pre-trained model to the novel data with a distribution shift using only a limited number of samples. Key challenges include how to identify and leverage the transferable knowledge learned by the pre-trained model. We therefore propose CDTD, or Causal Domain-Invariant Temporal Dynamics for knowledge transfer. To identify the temporally invariant and variant representations, we employ the causal representation learning methods for unsupervised pertaining, and then tune the classifier with supervisions in next stage. Specifically, we assume the domain information can be well estimated and the pre-trained image decoder and transition models can be well transferred. During adaptation, we fix the transferable temporal dynamics and update the image encoder and domain estimator. The efficacy of our approach is revealed by the superior accuracy of CDTD over leading alternatives across standard few-shot action recognition datasets.

Related papers

• Learn from the Learnt: Source-Free Active Domain Adaptation via Contrastive Sampling and Visual Persistence [60.37934652213881]
  Domain Adaptation (DA) facilitates knowledge transfer from a source domain to a related target domain. This paper investigates a practical DA paradigm, namely Source data-Free Active Domain Adaptation (SFADA), where source data becomes inaccessible during adaptation. We present learn from the learnt (LFTL), a novel paradigm for SFADA to leverage the learnt knowledge from the source pretrained model and actively iterated models without extra overhead.
  arXiv  Detail & Related papers  (2024-07-26T17:51:58Z)
• Learning Transferable Conceptual Prototypes for Interpretable Unsupervised Domain Adaptation [79.22678026708134]
  In this paper, we propose an inherently interpretable method, named Transferable Prototype Learning ( TCPL) To achieve this goal, we design a hierarchically prototypical module that transfers categorical basic concepts from the source domain to the target domain and learns domain-shared prototypes for explaining the underlying reasoning process. Comprehensive experiments show that the proposed method can not only provide effective and intuitive explanations but also outperform previous state-of-the-arts.
  arXiv  Detail & Related papers  (2023-10-12T06:36:41Z)
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation [48.039156140237615]
  A Continual Test-Time Adaptation task is proposed to adapt the pre-trained model to continually changing target domains. We design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain-specific and domain-shared knowledge. Our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks.
  arXiv  Detail & Related papers  (2023-06-07T11:18:53Z)
• Visualizing Transferred Knowledge: An Interpretive Model of Unsupervised Domain Adaptation [70.85686267987744]
  Unsupervised domain adaptation problems can transfer knowledge from a labeled source domain to an unlabeled target domain. We propose an interpretive model of unsupervised domain adaptation, as the first attempt to visually unveil the mystery of transferred knowledge. Our method provides an intuitive explanation for the base model's predictions and unveils transfer knowledge by matching the image patches with the same semantics across both source and target domains.
  arXiv  Detail & Related papers  (2023-03-04T03:02:12Z)
• Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation [14.473807945791132]
  Continual Test-Time Adaptation (CTTA) aims to adapt the source model to continually changing unlabeled target domains without access to the source data. Motivated by the prompt learning in NLP, in this paper we propose to learn an image-level visual domain prompt for target domains while having the source model parameters frozen.
  arXiv  Detail & Related papers  (2022-12-08T08:56:02Z)
• Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain [25.84756140221655]
  ConSecutive PreTraining (CSPT) is proposed based on the idea of not stopping pretraining in natural language processing (NLP) The proposed CSPT also can release the huge potential of unlabeled data for task-aware model training. The results show that by utilizing the proposed CSPT for task-aware model training, almost all downstream tasks in RSD can outperform the previous method of supervised pretraining-then-fine-tuning.
  arXiv  Detail & Related papers  (2022-07-08T12:32:09Z)
• Variational Transfer Learning using Cross-Domain Latent Modulation [1.9662978733004601]
  We introduce a novel cross-domain latent modulation mechanism to a variational autoencoder framework so as to achieve effective transfer learning. Deep representations of the source and target domains are first extracted by a unified inference model and aligned by employing gradient reversal. The learned deep representations are then cross-modulated to the latent encoding of the alternative domain, where consistency constraints are also applied.
  arXiv  Detail & Related papers  (2022-05-31T03:47:08Z)
• Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification [53.735029033681435]
  Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains. In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
  arXiv  Detail & Related papers  (2020-07-11T22:48:42Z)
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers [138.68213707587822]
  We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning. We show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function. Our approach is applicable to domains with continuous states and actions and does not require learning an explicit model of the dynamics.
  arXiv  Detail & Related papers  (2020-06-24T17:47:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.