Multi-Scale Neighborhood Occupancy Masked Autoencoder for Self-Supervised Learning in LiDAR Point Clouds

• URL: http://arxiv.org/abs/2502.20316v1
• Date: Thu, 27 Feb 2025 17:42:47 GMT
• Title: Multi-Scale Neighborhood Occupancy Masked Autoencoder for Self-Supervised Learning in LiDAR Point Clouds
• Authors: Mohamed Abdelsamad, Michael Ulrich, Claudius Gläser, Abhinav Valada,
• Abstract summary: Masked autoencoders (MAE) have shown tremendous potential for self-supervised learning (SSL) in vision and beyond.<n>Point clouds from LiDARs used in automated driving are particularly challenging for MAEs since large areas of the 3D volume are empty.<n>We propose the novel neighborhood occupancy MAE (NOMAE) that overcomes the aforementioned challenges by employing masked occupancy reconstruction only in the neighborhood of non-masked voxels.
• Score: 9.994719163112416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked autoencoders (MAE) have shown tremendous potential for self-supervised learning (SSL) in vision and beyond. However, point clouds from LiDARs used in automated driving are particularly challenging for MAEs since large areas of the 3D volume are empty. Consequently, existing work suffers from leaking occupancy information into the decoder and has significant computational complexity, thereby limiting the SSL pre-training to only 2D bird's eye view encoders in practice. In this work, we propose the novel neighborhood occupancy MAE (NOMAE) that overcomes the aforementioned challenges by employing masked occupancy reconstruction only in the neighborhood of non-masked voxels. We incorporate voxel masking and occupancy reconstruction at multiple scales with our proposed hierarchical mask generation technique to capture features of objects of different sizes in the point cloud. NOMAEs are extremely flexible and can be directly employed for SSL in existing 3D architectures. We perform extensive evaluations on the nuScenes and Waymo Open datasets for the downstream perception tasks of semantic segmentation and 3D object detection, comparing with both discriminative and generative SSL methods. The results demonstrate that NOMAE sets the new state-of-the-art on multiple benchmarks for multiple point cloud perception tasks.

Related papers

• PSA-SSL: Pose and Size-aware Self-Supervised Learning on LiDAR Point Clouds [8.645078288584305]
  We propose PSA-SSL, a novel extension to point cloud SSL that learns object pose and size-aware features. Our approach outperforms other state-of-the-art SSL methods on 3D semantic segmentation and 3D object detection.
  arXiv  Detail & Related papers  (2025-03-18T05:17:06Z)
• Triple Point Masking [49.39218611030084]
  Existing 3D mask learning methods encounter performance bottlenecks under limited data. We introduce a triple point masking scheme, named TPM, which serves as a scalable framework for pre-training of masked autoencoders. Extensive experiments show that the four baselines equipped with the proposed TPM achieve comprehensive performance improvements on various downstream tasks.
  arXiv  Detail & Related papers  (2024-09-26T05:33:30Z)
• Point Cloud Self-supervised Learning via 3D to Multi-view Masked Autoencoder [21.73287941143304]
  Multi-Modality Masked AutoEncoders (MAE) methods leverage both 2D images and 3D point clouds for pre-training. We introduce a novel approach employing a 3D to multi-view masked autoencoder to fully harness the multi-modal attributes of 3D point clouds. Our method outperforms state-of-the-art counterparts by a large margin in a variety of downstream tasks.
  arXiv  Detail & Related papers  (2023-11-17T22:10:03Z)
• MAELi: Masked Autoencoder for Large-Scale LiDAR Point Clouds [13.426810473131642]
  Masked AutoEncoder for LiDAR point clouds (MAELi) intuitively leverages the sparsity of LiDAR point clouds in both the encoder and decoder during reconstruction. In a novel reconstruction approach, MAELi distinguishes between empty and occluded space. Thereby, without any ground truth whatsoever and trained on single frames only, MAELi obtains an understanding of the underlying 3D scene geometry and semantics.
  arXiv  Detail & Related papers  (2022-12-14T13:10:27Z)
• GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds [72.60362979456035]
  Masked Autoencoders (MAE) are challenging to explore in large-scale 3D point clouds. We propose a textbfGenerative textbfDecoder for MAE (GD-MAE) to automatically merges the surrounding context. We demonstrate the efficacy of the proposed method on several large-scale benchmarks: KITTI, and ONCE.
  arXiv  Detail & Related papers  (2022-12-06T14:32:55Z)
• MAPLE: Masked Pseudo-Labeling autoEncoder for Semi-supervised Point Cloud Action Recognition [160.49403075559158]
  We propose a Masked Pseudo-Labeling autoEncoder (textbfMAPLE) framework for point cloud action recognition. In particular, we design a novel and efficient textbfDecoupled textbfspatial-textbftemporal TranstextbfFormer (textbfDestFormer) as the backbone of MAPLE. MAPLE achieves superior results on three public benchmarks and outperforms the state-of-the-art method by 8.08% accuracy on the MSR-Action3
  arXiv  Detail & Related papers  (2022-09-01T12:32:40Z)
• A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond [64.85076239939336]
  Self-supervised learning (SSL) in vision might undertake a similar trajectory as in NLP. generative pretext tasks with the masked prediction (e.g., BERT) have become a de facto standard SSL practice in NLP. Success of mask image modeling has revived the masking autoencoder.
  arXiv  Detail & Related papers  (2022-07-30T09:59:28Z)
• Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoders [13.119676419877244]
  We propose a solution to reduce the dependence on labelled 3D training data by leveraging pre-training on large-scale unlabeled outdoor LiDAR point clouds. Our approach introduces a new self-supervised masked occupancy pre-training method called Occupancy-MAE. For 3D object detection, Occupancy-MAE reduces the labelled data required for car detection on the KITTI dataset by half. For 3D semantic segmentation, Occupancy-MAE outperforms training from scratch by around 2% in mIoU.
  arXiv  Detail & Related papers  (2022-06-20T17:15:50Z)
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training [56.81809311892475]
  Masked Autoencoders (MAE) have shown great potentials in self-supervised pre-training for language and 2D image transformers. We propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds.
  arXiv  Detail & Related papers  (2022-05-28T11:22:53Z)
• Self-Supervised Point Cloud Representation Learning with Occlusion Auto-Encoder [63.77257588569852]
  We present 3D Occlusion Auto-Encoder (3D-OAE) for learning representations for point clouds. Our key idea is to randomly occlude some local patches of the input point cloud and establish the supervision via recovering the occluded patches. In contrast with previous methods, our 3D-OAE can remove a large proportion of patches and predict them only with a small number of visible patches.
  arXiv  Detail & Related papers  (2022-03-26T14:06:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.