Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysis

• URL: http://arxiv.org/abs/2210.15904v1
• Date: Fri, 28 Oct 2022 05:23:03 GMT
• Title: Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysis
• Authors: Bach Tran, Binh-Son Hua, Anh Tuan Tran, Minh Hoai
• Abstract summary: We propose a novel pre-training method for 3D point cloud models. Our pre-training is self-supervised by a local pixel/point level correspondence loss and a global image/point cloud level loss. These improved models outperform existing state-of-the-art methods on various datasets and downstream tasks.
• Score: 33.31864436614945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, great progress has been made in 3D deep learning with the emergence of deep neural networks specifically designed for 3D point clouds. These networks are often trained from scratch or from pre-trained models learned purely from point cloud data. Inspired by the success of deep learning in the image domain, we devise a novel pre-training technique for better model initialization by utilizing the multi-view rendering of the 3D data. Our pre-training is self-supervised by a local pixel/point level correspondence loss computed from perspective projection and a global image/point cloud level loss based on knowledge distillation, thus effectively improving upon popular point cloud networks, including PointNet, DGCNN and SR-UNet. These improved models outperform existing state-of-the-art methods on various datasets and downstream tasks. We also analyze the benefits of synthetic and real data for pre-training, and observe that pre-training on synthetic data is also useful for high-level downstream tasks. Code and pre-trained models are available at https://github.com/VinAIResearch/selfsup_pcd.

Related papers

• GS-PT: Exploiting 3D Gaussian Splatting for Comprehensive Point Cloud Understanding via Self-supervised Learning [15.559369116540097]
  Self-supervised learning of point cloud aims to leverage unlabeled 3D data to learn meaningful representations without reliance on manual annotations. We propose GS-PT, which integrates 3D Gaussian Splatting (3DGS) into point cloud self-supervised learning for the first time. Our pipeline utilizes transformers as the backbone for self-supervised pre-training and introduces novel contrastive learning tasks through 3DGS.
  arXiv  Detail & Related papers  (2024-09-08T03:46:47Z)
• PointRegGPT: Boosting 3D Point Cloud Registration using Generative Point-Cloud Pairs for Training [90.06520673092702]
  We present PointRegGPT, boosting 3D point cloud registration using generative point-cloud pairs for training. To our knowledge, this is the first generative approach that explores realistic data generation for indoor point cloud registration.
  arXiv  Detail & Related papers  (2024-07-19T06:29:57Z)
• AutoSynth: Learning to Generate 3D Training Data for Object Point Cloud Registration [69.21282992341007]
  Auto Synth automatically generates 3D training data for point cloud registration. We replace the point cloud registration network with a much smaller surrogate network, leading to a $4056.43$ speedup. Our results on TUD-L, LINEMOD and Occluded-LINEMOD evidence that a neural network trained on our searched dataset yields consistently better performance than the same one trained on the widely used ModelNet40 dataset.
  arXiv  Detail & Related papers  (2023-09-20T09:29:44Z)
• Ponder: Point Cloud Pre-training via Neural Rendering [93.34522605321514]
  We propose a novel approach to self-supervised learning of point cloud representations by differentiable neural encoders. The learned point-cloud can be easily integrated into various downstream tasks, including not only high-level rendering tasks like 3D detection and segmentation, but low-level tasks like 3D reconstruction and image rendering.
  arXiv  Detail & Related papers  (2022-12-31T08:58:39Z)
• 3D Point Cloud Pre-training with Knowledge Distillation from 2D Images [128.40422211090078]
  We propose a knowledge distillation method for 3D point cloud pre-trained models to acquire knowledge directly from the 2D representation learning model. Specifically, we introduce a cross-attention mechanism to extract concept features from 3D point cloud and compare them with the semantic information from 2D images. In this scheme, the point cloud pre-trained models learn directly from rich information contained in 2D teacher models.
  arXiv  Detail & Related papers  (2022-12-17T23:21:04Z)
• P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting [94.11915008006483]
  We propose a novel Point-to-Pixel prompting for point cloud analysis. Our method attains 89.3% accuracy on the hardest setting of ScanObjectNN. Our framework also exhibits very competitive performance on ModelNet classification and ShapeNet Part Code.
  arXiv  Detail & Related papers  (2022-08-04T17:59:03Z)
• Continual learning on 3D point clouds with random compressed rehearsal [10.667104977730304]
  This work proposes a novel neural network architecture capable of continual learning on 3D point cloud data. We utilize point cloud structure properties for preserving a heavily compressed set of past data.
  arXiv  Detail & Related papers  (2022-05-16T22:59:52Z)
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding [107.02479689909164]
  In this work, we aim at facilitating research on 3D representation learning. We measure the effect of unsupervised pre-training on a large source set of 3D scenes.
  arXiv  Detail & Related papers  (2020-07-21T17:59:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.