SAM4UDASS: When SAM Meets Unsupervised Domain Adaptive Semantic Segmentation in Intelligent Vehicles

• URL: http://arxiv.org/abs/2401.08604v1
• Date: Wed, 22 Nov 2023 08:29:45 GMT
• Title: SAM4UDASS: When SAM Meets Unsupervised Domain Adaptive Semantic Segmentation in Intelligent Vehicles
• Authors: Weihao Yan, Yeqiang Qian, Xingyuan Chen, Hanyang Zhuang, Chunxiang Wang, Ming Yang
• Abstract summary: We introduce SAM4UDASS, a novel approach that incorporates the Segment Anything Model (SAM) into self-training UDA methods for refining pseudo-labels. It involves Semantic-Guided Mask Labeling, which assigns semantic labels to unlabeled SAM masks using UDA pseudo-labels. It brings more than 3% mIoU gains on GTA5-to-Cityscapes, SYNTHIA-to-Cityscapes, and Cityscapes-to-ACDC when using DAFormer and achieves SOTA when using MIC.
• Score: 27.405213492173186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic segmentation plays a critical role in enabling intelligent vehicles to comprehend their surrounding environments. However, deep learning-based methods usually perform poorly in domain shift scenarios due to the lack of labeled data for training. Unsupervised domain adaptation (UDA) techniques have emerged to bridge the gap across different driving scenes and enhance model performance on unlabeled target environments. Although self-training UDA methods have achieved state-of-the-art results, the challenge of generating precise pseudo-labels persists. These pseudo-labels tend to favor majority classes, consequently sacrificing the performance of rare classes or small objects like traffic lights and signs. To address this challenge, we introduce SAM4UDASS, a novel approach that incorporates the Segment Anything Model (SAM) into self-training UDA methods for refining pseudo-labels. It involves Semantic-Guided Mask Labeling, which assigns semantic labels to unlabeled SAM masks using UDA pseudo-labels. Furthermore, we devise fusion strategies aimed at mitigating semantic granularity inconsistency between SAM masks and the target domain. SAM4UDASS innovatively integrate SAM with UDA for semantic segmentation in driving scenes and seamlessly complements existing self-training UDA methodologies. Extensive experiments on synthetic-to-real and normal-to-adverse driving datasets demonstrate its effectiveness. It brings more than 3% mIoU gains on GTA5-to-Cityscapes, SYNTHIA-to-Cityscapes, and Cityscapes-to-ACDC when using DAFormer and achieves SOTA when using MIC. The code will be available at https://github.com/ywher/SAM4UDASS.

Related papers

• PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Images [16.662173255725463]
  We propose a novel Pointly-supervised Segment Anything Model named PointSAM. We conduct experiments on RSI datasets, including WHU, HRSID, and NWPU VHR-10. The results show that our method significantly outperforms direct testing with SAM, SAM2, and other comparison methods.
  arXiv  Detail & Related papers  (2024-09-20T11:02:18Z)
• SS-ADA: A Semi-Supervised Active Domain Adaptation Framework for Semantic Segmentation [25.929173344653158]
  We propose a novel semi-supervised active domain adaptation (SS-ADA) framework for semantic segmentation. SS-ADA integrates active learning into semi-supervised semantic segmentation to achieve the accuracy of supervised learning. We conducted extensive experiments on synthetic-to-real and real-to-real domain adaptation settings.
  arXiv  Detail & Related papers  (2024-06-17T13:40:42Z)
• ECAP: Extensive Cut-and-Paste Augmentation for Unsupervised Domain Adaptive Semantic Segmentation [4.082799056366928]
  We propose an extensive cut-and-paste strategy (ECAP) to leverage reliable pseudo-labels through data augmentation. ECAP maintains a memory bank of pseudo-labeled target samples throughout training and cut-and-pastes the most confident ones onto the current training batch. We implement ECAP on top of the recent method MIC and boost its performance on two synthetic-to-real domain adaptation benchmarks.
  arXiv  Detail & Related papers  (2024-03-06T17:06:07Z)
• MoDA: Leveraging Motion Priors from Videos for Advancing Unsupervised Domain Adaptation in Semantic Segmentation [61.4598392934287]
  This study introduces a different UDA scenarios where the target domain contains unlabeled video frames. We design a textbfMotion-guided textbfDomain textbfAdaptive semantic segmentation framework (MoDA) MoDA harnesses the self-supervised object motion cues to facilitate cross-domain alignment for segmentation task.
  arXiv  Detail & Related papers  (2023-09-21T01:31:54Z)
• Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation [30.812323329239614]
  Weakly supervised semantic segmentation (WSSS) aims to bypass the need for laborious pixel-level annotation by using only image-level annotation. Most existing methods rely on Class Activation Maps (CAM) to derive pixel-level pseudo-labels. We introduce a simple yet effective method harnessing the Segment Anything Model (SAM), a class-agnostic foundation model capable of producing fine-grained instance masks of objects, parts, and subparts.
  arXiv  Detail & Related papers  (2023-05-09T23:24:09Z)
• Hard-aware Instance Adaptive Self-training for Unsupervised Cross-domain Semantic Segmentation [18.807921765977415]
  We propose a hard-aware instance adaptive self-training framework for UDA on the task of semantic segmentation. We develop a novel pseudo-label generation strategy with an instance adaptive selector. Experiments on GTA5 to Cityscapes, SYNTHIA to Cityscapes, and Cityscapes to Oxford RobotCar demonstrate the superior performance of our approach.
  arXiv  Detail & Related papers  (2023-02-14T11:52:26Z)
• MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation [104.40114562948428]
  In unsupervised domain adaptation (UDA), a model trained on source data (e.g. synthetic) is adapted to target data (e.g. real-world) without access to target annotation. We propose a Masked Image Consistency (MIC) module to enhance UDA by learning spatial context relations of the target domain. MIC significantly improves the state-of-the-art performance across the different recognition tasks for synthetic-to-real, day-to-nighttime, and clear-to-adverse-weather UDA.
  arXiv  Detail & Related papers  (2022-12-02T17:29:32Z)
• Exploiting Instance-based Mixed Sampling via Auxiliary Source Domain Supervision for Domain-adaptive Action Detection [75.38704117155909]
  We propose a novel domain adaptive action detection approach and a new adaptation protocol. Self-training combined with cross-domain mixed sampling has shown remarkable performance gain in UDA context. We name our proposed framework as domain-adaptive action instance mixing (DA-AIM)
  arXiv  Detail & Related papers  (2022-09-28T22:03:25Z)
• Semi-Supervised Domain Adaptation with Prototypical Alignment and Consistency Learning [86.6929930921905]
  This paper studies how much it can help address domain shifts if we further have a few target samples labeled. To explore the full potential of landmarks, we incorporate a prototypical alignment (PA) module which calculates a target prototype for each class from the landmarks. Specifically, we severely perturb the labeled images, making PA non-trivial to achieve and thus promoting model generalizability.
  arXiv  Detail & Related papers  (2021-04-19T08:46:08Z)
• Cycle Self-Training for Domain Adaptation [85.14659717421533]
  Cycle Self-Training (CST) is a principled self-training algorithm that enforces pseudo-labels to generalize across domains. CST recovers target ground truth, while both invariant feature learning and vanilla self-training fail. Empirical results indicate that CST significantly improves over prior state-of-the-arts in standard UDA benchmarks.
  arXiv  Detail & Related papers  (2021-03-05T10:04:25Z)
• Domain Adaptive Semantic Segmentation Using Weak Labels [115.16029641181669]
  We propose a novel framework for domain adaptation in semantic segmentation with image-level weak labels in the target domain. We develop a weak-label classification module to enforce the network to attend to certain categories. In experiments, we show considerable improvements with respect to the existing state-of-the-arts in UDA and present a new benchmark in the WDA setting.
  arXiv  Detail & Related papers  (2020-07-30T01:33:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.