Multi-Modal Continual Test-Time Adaptation for 3D Semantic Segmentation
- URL: http://arxiv.org/abs/2303.10457v1
- Date: Sat, 18 Mar 2023 16:51:19 GMT
- Title: Multi-Modal Continual Test-Time Adaptation for 3D Semantic Segmentation
- Authors: Haozhi Cao, Yuecong Xu, Jianfei Yang, Pengyu Yin, Shenghai Yuan, Lihua
Xie
- Abstract summary: Continual Test-Time Adaptation (CTTA) generalizes conventional Test-Time Adaptation (TTA) by assuming that the target domain is dynamic over time rather than stationary.
In this paper, we explore Multi-Modal Continual Test-Time Adaptation (MM-CTTA) as a new extension of CTTA for 3D semantic segmentation.
- Score: 26.674085603033742
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Continual Test-Time Adaptation (CTTA) generalizes conventional Test-Time
Adaptation (TTA) by assuming that the target domain is dynamic over time rather
than stationary. In this paper, we explore Multi-Modal Continual Test-Time
Adaptation (MM-CTTA) as a new extension of CTTA for 3D semantic segmentation.
The key to MM-CTTA is to adaptively attend to the reliable modality while
avoiding catastrophic forgetting during continual domain shifts, which is out
of the capability of previous TTA or CTTA methods. To fulfill this gap, we
propose an MM-CTTA method called Continual Cross-Modal Adaptive Clustering
(CoMAC) that addresses this task from two perspectives. On one hand, we propose
an adaptive dual-stage mechanism to generate reliable cross-modal predictions
by attending to the reliable modality based on the class-wise feature-centroid
distance in the latent space. On the other hand, to perform test-time
adaptation without catastrophic forgetting, we design class-wise momentum
queues that capture confident target features for adaptation while
stochastically restoring pseudo-source features to revisit source knowledge. We
further introduce two new benchmarks to facilitate the exploration of MM-CTTA
in the future. Our experimental results show that our method achieves
state-of-the-art performance on both benchmarks.
Related papers
- Adaptive Cascading Network for Continual Test-Time Adaptation [12.718826132518577]
We study the problem of continual test-time adaption where the goal is to adapt a source pre-trained model to a sequence of unlabelled target domains at test time.
Existing methods on test-time training suffer from several limitations.
arXiv Detail & Related papers (2024-07-17T01:12:57Z) - Exploring Test-Time Adaptation for Object Detection in Continually Changing Environments [13.163784646113214]
Continual Test-Time Adaptation (CTTA) has emerged as a promising technique to gradually adapt a source-trained model to test data drawn from a continually changing target domain.
We present CTAOD, aiming to enhance performance of detection models in CTTA scenarios.
We demonstrate the effectiveness of our approach on four CTTA tasks for object detection, where CTAOD outperforms existing methods.
arXiv Detail & Related papers (2024-06-24T08:30:03Z) - Unleashing Network Potentials for Semantic Scene Completion [50.95486458217653]
This paper proposes a novel SSC framework - Adrial Modality Modulation Network (AMMNet)
AMMNet introduces two core modules: a cross-modal modulation enabling the interdependence of gradient flows between modalities, and a customized adversarial training scheme leveraging dynamic gradient competition.
Extensive experimental results demonstrate that AMMNet outperforms state-of-the-art SSC methods by a large margin.
arXiv Detail & Related papers (2024-03-12T11:48:49Z) - Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation [33.75630514826721]
We propose a distribution-aware tuning ( DAT) method to make semantic segmentation CTTA efficient and practical in real-world applications.
DAT adaptively selects and updates two small groups of trainable parameters based on data distribution during the continual adaptation process.
We conduct experiments on two widely-used semantic segmentation CTTA benchmarks, achieving promising performance compared to previous state-of-the-art methods.
arXiv Detail & Related papers (2023-09-24T10:48:20Z) - AR-TTA: A Simple Method for Real-World Continual Test-Time Adaptation [16.85284386728494]
We propose to validate test-time adaptation methods using datasets for autonomous driving, namely CLAD-C and SHIFT.
We observe that current test-time adaptation methods struggle to effectively handle varying degrees of domain shift.
The proposed method, named AR-TTA, outperforms existing approaches on both synthetic and more real-world benchmarks.
arXiv Detail & Related papers (2023-09-18T19:34:23Z) - Uncertainty Guided Adaptive Warping for Robust and Efficient Stereo
Matching [77.133400999703]
Correlation based stereo matching has achieved outstanding performance.
Current methods with a fixed model do not work uniformly well across various datasets.
This paper proposes a new perspective to dynamically calculate correlation for robust stereo matching.
arXiv Detail & Related papers (2023-07-26T09:47:37Z) - On Pitfalls of Test-Time Adaptation [82.8392232222119]
Test-Time Adaptation (TTA) has emerged as a promising approach for tackling the robustness challenge under distribution shifts.
We present TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art algorithms, a diverse array of distribution shifts, and two evaluation protocols.
arXiv Detail & Related papers (2023-06-06T09:35:29Z) - DLTTA: Dynamic Learning Rate for Test-time Adaptation on Cross-domain
Medical Images [56.72015587067494]
We propose a novel dynamic learning rate adjustment method for test-time adaptation, called DLTTA.
Our method achieves effective and fast test-time adaptation with consistent performance improvement over current state-of-the-art test-time adaptation methods.
arXiv Detail & Related papers (2022-05-27T02:34:32Z) - MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation [104.48766162008815]
We propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation.
To design a framework that can take full advantage of multi-modality, each modality provides regularized self-supervisory signals to other modalities.
Our regularized pseudo labels produce stable self-learning signals in numerous multi-modal test-time adaptation scenarios.
arXiv Detail & Related papers (2022-04-27T02:28:12Z) - A Transductive Multi-Head Model for Cross-Domain Few-Shot Learning [72.30054522048553]
We present a new method, Transductive Multi-Head Few-Shot learning (TMHFS), to address the Cross-Domain Few-Shot Learning challenge.
The proposed methods greatly outperform the strong baseline, fine-tuning, on four different target domains.
arXiv Detail & Related papers (2020-06-08T02:39:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.