Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-time Adaptation

• URL: http://arxiv.org/abs/2408.13983v1
• Date: Mon, 26 Aug 2024 02:33:47 GMT
• Title: Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-time Adaptation
• Authors: Yushun Tang, Shuoshuo Chen, Zhihe Lu, Xinchao Wang, Zhihai He,
• Abstract summary: We introduce a dual-path token lifting for domain shift correction in test time adaptation. We then perform dual-path lifting with interleaved token prediction and update between the path of domain shift tokens and the path of class tokens. Experimental results on the benchmark datasets demonstrate that our proposed method significantly improves the online fully test-time domain adaptation performance.
• Score: 59.18151483767509
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based methods have achieved remarkable success in various machine learning tasks. How to design efficient test-time adaptation methods for transformer models becomes an important research task. In this work, motivated by the dual-subband wavelet lifting scheme developed in multi-scale signal processing which is able to efficiently separate the input signals into principal components and noise components, we introduce a dual-path token lifting for domain shift correction in test time adaptation. Specifically, we introduce an extra token, referred to as \textit{domain shift token}, at each layer of the transformer network. We then perform dual-path lifting with interleaved token prediction and update between the path of domain shift tokens and the path of class tokens at all network layers. The prediction and update networks are learned in an adversarial manner. Specifically, the task of the prediction network is to learn the residual noise of domain shift which should be largely invariant across all classes and all samples in the target domain. In other words, the predicted domain shift noise should be indistinguishable between all sample classes. On the other hand, the task of the update network is to update the class tokens by removing the domain shift from the input image samples so that input samples become more discriminative between different classes in the feature space. To effectively learn the prediction and update networks with two adversarial tasks, both theoretically and practically, we demonstrate that it is necessary to use smooth optimization for the update network but non-smooth optimization for the prediction network. Experimental results on the benchmark datasets demonstrate that our proposed method significantly improves the online fully test-time domain adaptation performance. Code is available at \url{https://github.com/yushuntang/DPAL}.

Related papers

• Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation [24.294049653744185]
  In transformer-based image classification, the class token at the first transformer encoder layer can be learned to capture the domain-specific characteristics of target samples during test-time adaptation. We propose a bi-level learning approach to capture the long-term variations of domain-specific characteristics while accommodating local variations of instance-specific characteristics. Our proposed bi-level visual conditioning token learning method is able to achieve significantly improved test-time adaptation performance by up to 1.9%.
  arXiv  Detail & Related papers  (2024-06-27T17:16:23Z)
• CCF: Cross Correcting Framework for Pedestrian Trajectory Prediction [7.9449756510822915]
  We propose a Cross-Correction Framework (CCF) to learn representations of pedestrian trajectories better. CCF consists of two prediction models which are trained with both cross-temporal loss and trajectory prediction loss. We utilize transformer-based encoder-correction-decoder architecture each to capture motion and social interaction among pedestrians.
  arXiv  Detail & Related papers  (2024-06-02T14:07:13Z)
• Cross-Domain Few-Shot Learning via Adaptive Transformer Networks [16.289485655725013]
  This paper proposes an adaptive transformer network (ADAPTER) for cross-domain few-shot learning. ADAPTER is built upon the idea of bidirectional cross-attention to learn transferable features between the two domains.
  arXiv  Detail & Related papers  (2024-01-25T07:05:42Z)
• Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback [12.946419909506883]
  We create a closed-loop system that makes use of a test-time feedback signal to adapt a network on the fly. We show that this loop can be effectively implemented using a learning-based function, which realizes an amortized for the network. This leads to an adaptation method, named Rapid Network Adaptation (RNA), that is notably more flexible and orders of magnitude faster than the baselines.
  arXiv  Detail & Related papers  (2023-09-27T16:20:39Z)
• Unsupervised Domain-adaptive Hash for Networks [81.49184987430333]
  Domain-adaptive hash learning has enjoyed considerable success in the computer vision community. We develop an unsupervised domain-adaptive hash learning method for networks, dubbed UDAH.
  arXiv  Detail & Related papers  (2021-08-20T12:09:38Z)
• Adversarial Bipartite Graph Learning for Video Domain Adaptation [50.68420708387015]
  Domain adaptation techniques, which focus on adapting models between distributionally different domains, are rarely explored in the video recognition area. Recent works on visual domain adaptation which leverage adversarial learning to unify the source and target video representations are not highly effective on the videos. This paper proposes an Adversarial Bipartite Graph (ABG) learning framework which directly models the source-target interactions.
  arXiv  Detail & Related papers  (2020-07-31T03:48:41Z)
• Self domain adapted network [6.040230864736051]
  Domain shift is a major problem for deploying deep networks in clinical practice. We propose a novel self domain adapted network (SDA-Net) that can rapidly adapt itself to a single test subject.
  arXiv  Detail & Related papers  (2020-07-07T01:41:34Z)
• Adaptive Risk Minimization: Learning to Adapt to Domain Shift [109.87561509436016]
  A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution. In this work, we consider the problem setting of domain generalization, where the training data are structured into domains and there may be multiple test time shifts. We introduce the framework of adaptive risk minimization (ARM), in which models are directly optimized for effective adaptation to shift by learning to adapt on the training domains.
  arXiv  Detail & Related papers  (2020-07-06T17:59:30Z)
• Few-Shot Learning as Domain Adaptation: Algorithm and Analysis [120.75020271706978]
  Few-shot learning uses prior knowledge learned from the seen classes to recognize the unseen classes. This class-difference-caused distribution shift can be considered as a special case of domain shift. We propose a prototypical domain adaptation network with attention (DAPNA) to explicitly tackle such a domain shift problem in a meta-learning framework.
  arXiv  Detail & Related papers  (2020-02-06T01:04:53Z)
• Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks [95.51368472949308]
  Adaptation can be useful in cases when training data is scarce, or when one wishes to encode priors in the network. In this paper, we propose a straightforward alternative: side-tuning.
  arXiv  Detail & Related papers  (2019-12-31T18:52:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.