Learning to Adapt SAM for Segmenting Cross-domain Point Clouds
- URL: http://arxiv.org/abs/2310.08820v3
- Date: Thu, 14 Dec 2023 00:27:13 GMT
- Title: Learning to Adapt SAM for Segmenting Cross-domain Point Clouds
- Authors: Xidong Peng, Runnan Chen, Feng Qiao, Lingdong Kong, Youquan Liu, Tai
Wang, Xinge Zhu, Yuexin Ma
- Abstract summary: Unsupervised domain adaptation (UDA) in 3D segmentation tasks presents a formidable challenge.
We propose an innovative hybrid feature augmentation methodology, which significantly enhances the alignment between the 3D feature space and SAM's feature space.
Our method is evaluated on many widely-recognized datasets and achieves state-of-the-art performance.
- Score: 26.58008890592077
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Unsupervised domain adaptation (UDA) in 3D segmentation tasks presents a
formidable challenge, primarily stemming from the sparse and unordered nature
of point cloud data. Especially for LiDAR point clouds, the domain discrepancy
becomes obvious across varying capture scenes, fluctuating weather conditions,
and the diverse array of LiDAR devices in use. While previous UDA methodologies
have often sought to mitigate this gap by aligning features between source and
target domains, this approach falls short when applied to 3D segmentation due
to the substantial domain variations. Inspired by the remarkable generalization
capabilities exhibited by the vision foundation model, SAM, in the realm of
image segmentation, our approach leverages the wealth of general knowledge
embedded within SAM to unify feature representations across diverse 3D domains
and further solves the 3D domain adaptation problem. Specifically, we harness
the corresponding images associated with point clouds to facilitate knowledge
transfer and propose an innovative hybrid feature augmentation methodology,
which significantly enhances the alignment between the 3D feature space and
SAM's feature space, operating at both the scene and instance levels. Our
method is evaluated on many widely-recognized datasets and achieves
state-of-the-art performance.
Related papers
- Visual Foundation Models Boost Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation [17.875516787157018]
We study how to harness the knowledge priors learned by 2D visual foundation models to produce more accurate labels for unlabeled target domains.
Our method is evaluated on various autonomous driving datasets and the results demonstrate a significant improvement for 3D segmentation task.
arXiv Detail & Related papers (2024-03-15T03:58:17Z) - CMDA: Cross-Modal and Domain Adversarial Adaptation for LiDAR-Based 3D
Object Detection [14.063365469339812]
LiDAR-based 3D Object Detection methods often do not generalize well to target domains outside the source (or training) data distribution.
We introduce a novel unsupervised domain adaptation (UDA) method, called CMDA, which leverages visual semantic cues from an image modality.
We also introduce a self-training-based learning strategy, wherein a model is adversarially trained to generate domain-invariant features.
arXiv Detail & Related papers (2024-03-06T14:12:38Z) - InvariantOODG: Learning Invariant Features of Point Clouds for
Out-of-Distribution Generalization [17.96808017359983]
We propose InvariantOODG, which learns invariability between point clouds with different distributions.
We define a set of learnable anchor points that locate the most useful local regions and two types of transformations to augment the input point clouds.
The experimental results demonstrate the effectiveness of the proposed model on 3D domain generalization benchmarks.
arXiv Detail & Related papers (2024-01-08T09:41:22Z) - Improving Anomaly Segmentation with Multi-Granularity Cross-Domain
Alignment [17.086123737443714]
Anomaly segmentation plays a pivotal role in identifying atypical objects in images, crucial for hazard detection in autonomous driving systems.
While existing methods demonstrate noteworthy results on synthetic data, they often fail to consider the disparity between synthetic and real-world data domains.
We introduce the Multi-Granularity Cross-Domain Alignment framework, tailored to harmonize features across domains at both the scene and individual sample levels.
arXiv Detail & Related papers (2023-08-16T22:54:49Z) - BEV-DG: Cross-Modal Learning under Bird's-Eye View for Domain
Generalization of 3D Semantic Segmentation [59.99683295806698]
Cross-modal Unsupervised Domain Adaptation (UDA) aims to exploit the complementarity of 2D-3D data to overcome the lack of annotation in a new domain.
We propose cross-modal learning under bird's-eye view for Domain Generalization (DG) of 3D semantic segmentation, called BEV-DG.
arXiv Detail & Related papers (2023-08-12T11:09:17Z) - SUG: Single-dataset Unified Generalization for 3D Point Cloud
Classification [44.27324696068285]
We propose a Single-dataset Unified Generalization (SUG) framework to alleviate the unforeseen domain differences faced by a well-trained source model.
Specifically, we first design a Multi-grained Sub-domain Alignment (MSA) method, which can constrain the learned representations to be domain-agnostic and discriminative.
Then, a Sample-level Domain-aware Attention (SDA) strategy is presented, which can selectively enhance easy-to-adapt samples from different sub-domains.
arXiv Detail & Related papers (2023-05-16T04:36:04Z) - AGO-Net: Association-Guided 3D Point Cloud Object Detection Network [86.10213302724085]
We propose a novel 3D detection framework that associates intact features for objects via domain adaptation.
We achieve new state-of-the-art performance on the KITTI 3D detection benchmark in both accuracy and speed.
arXiv Detail & Related papers (2022-08-24T16:54:38Z) - Unsupervised Domain Adaptation for Monocular 3D Object Detection via
Self-Training [57.25828870799331]
We propose STMono3D, a new self-teaching framework for unsupervised domain adaptation on Mono3D.
We develop a teacher-student paradigm to generate adaptive pseudo labels on the target domain.
STMono3D achieves remarkable performance on all evaluated datasets and even surpasses fully supervised results on the KITTI 3D object detection dataset.
arXiv Detail & Related papers (2022-04-25T12:23:07Z) - Unsupervised Domain Adaptive 3D Detection with Multi-Level Consistency [90.71745178767203]
Deep learning-based 3D object detection has achieved unprecedented success with the advent of large-scale autonomous driving datasets.
Existing 3D domain adaptive detection methods often assume prior access to the target domain annotations, which is rarely feasible in the real world.
We study a more realistic setting, unsupervised 3D domain adaptive detection, which only utilizes source domain annotations.
arXiv Detail & Related papers (2021-07-23T17:19:23Z) - AFAN: Augmented Feature Alignment Network for Cross-Domain Object
Detection [90.18752912204778]
Unsupervised domain adaptation for object detection is a challenging problem with many real-world applications.
We propose a novel augmented feature alignment network (AFAN) which integrates intermediate domain image generation and domain-adversarial training.
Our approach significantly outperforms the state-of-the-art methods on standard benchmarks for both similar and dissimilar domain adaptations.
arXiv Detail & Related papers (2021-06-10T05:01:20Z) - Domain Conditioned Adaptation Network [90.63261870610211]
We propose a Domain Conditioned Adaptation Network (DCAN) to excite distinct convolutional channels with a domain conditioned channel attention mechanism.
This is the first work to explore the domain-wise convolutional channel activation for deep DA networks.
arXiv Detail & Related papers (2020-05-14T04:23:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.