DODA: Data-oriented Sim-to-Real Domain Adaptation for 3D Indoor Semantic Segmentation

• URL: http://arxiv.org/abs/2204.01599v1
• Date: Mon, 4 Apr 2022 15:52:55 GMT
• Title: DODA: Data-oriented Sim-to-Real Domain Adaptation for 3D Indoor Semantic Segmentation
• Authors: Runyu Ding, Jihan Yang, Li Jiang, Xiaojuan Qi
• Abstract summary: We propose a Data-Oriented Domain Adaptation (DODA) framework to mitigate pattern and context gaps. Our DODA encompasses virtual scan simulation to imitate real-world point cloud patterns and tail-aware cuboid mixing to alleviate the interior context gap. Our DODA surpasses existing UDA approaches by over 13% on both 3D-FRONT $rightarrow$ ScanNet and 3D-FRONT $rightarrow$ S3DIS.
• Score: 36.37396175140793
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning approaches achieve prominent success in 3D semantic segmentation. However, collecting densely annotated real-world 3D datasets is extremely time-consuming and expensive. Training models on synthetic data and generalizing on real-world scenarios becomes an appealing alternative, but unfortunately suffers from notorious domain shifts. In this work, we propose a Data-Oriented Domain Adaptation (DODA) framework to mitigate pattern and context gaps caused by different sensing mechanisms and layout placements across domains. Our DODA encompasses virtual scan simulation to imitate real-world point cloud patterns and tail-aware cuboid mixing to alleviate the interior context gap with a cuboid-based intermediate domain. The first unsupervised sim-to-real adaptation benchmark on 3D indoor semantic segmentation is also built on 3D-FRONT, ScanNet and S3DIS along with 7 popular Unsupervised Domain Adaptation (UDA) methods. Our DODA surpasses existing UDA approaches by over 13% on both 3D-FRONT $\rightarrow$ ScanNet and 3D-FRONT $\rightarrow$ S3DIS. Code will be available.

Related papers

• Syn-to-Real Unsupervised Domain Adaptation for Indoor 3D Object Detection [50.448520056844885]
  We propose a novel framework for syn-to-real unsupervised domain adaptation in indoor 3D object detection. Our adaptation results from synthetic dataset 3D-FRONT to real-world datasets ScanNetV2 and SUN RGB-D demonstrate remarkable mAP25 improvements of 9.7% and 9.1% over Source-Only baselines.
  arXiv  Detail & Related papers  (2024-06-17T08:18:41Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction [62.599588577671796]
  We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality.
  arXiv  Detail & Related papers  (2023-11-29T20:30:18Z)
• T-UDA: Temporal Unsupervised Domain Adaptation in Sequential Point Clouds [2.5291108878852864]
  unsupervised domain adaptation (UDA) methods adapt models trained on one (source) domain with annotations available to another (target) domain for which only unannotated data are available. We introduce a novel domain adaptation method that leverages the best of both trends. Dubbed T-UDA for "temporal UDA", such a combination yields massive performance gains for the task of 3D semantic segmentation of driving scenes.
  arXiv  Detail & Related papers  (2023-09-15T10:47:12Z)
• 3D Adversarial Augmentations for Robust Out-of-Domain Predictions [115.74319739738571]
  We focus on improving the generalization to out-of-domain data. We learn a set of vectors that deform the objects in an adversarial fashion. We perform adversarial augmentation by applying the learned sample-independent vectors to the available objects when training a model.
  arXiv  Detail & Related papers  (2023-08-29T17:58:55Z)
• SSDA3D: Semi-supervised Domain Adaptation for 3D Object Detection from Point Cloud [125.9472454212909]
  We present a novel Semi-Supervised Domain Adaptation method for 3D object detection (SSDA3D) SSDA3D includes an Inter-domain Adaptation stage and an Intra-domain Generalization stage. Experiments show that, with only 10% labeled target data, our SSDA3D can surpass the fully-supervised oracle model with 100% target label.
  arXiv  Detail & Related papers  (2022-12-06T09:32:44Z)
• Semi-supervised 3D shape segmentation with multilevel consistency and part substitution [21.075426681857024]
  We propose an effective semi-supervised method for learning 3D segmentations from a few labeled 3D shapes and a large amount of unlabeled 3D data. For the unlabeled data, we present a novel multilevel consistency loss to enforce consistency of network predictions between perturbed copies of a 3D shape. For the labeled data, we develop a simple yet effective part substitution scheme to augment the labeled 3D shapes with more structural variations to enhance training.
  arXiv  Detail & Related papers  (2022-04-19T11:48:24Z)
• 3D-Aware Indoor Scene Synthesis with Depth Priors [62.82867334012399]
  Existing methods fail to model indoor scenes due to the large diversity of room layouts and the objects inside. We argue that indoor scenes do not have a shared intrinsic structure, and hence only using 2D images cannot adequately guide the model with the 3D geometry.
  arXiv  Detail & Related papers  (2022-02-17T09:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.