P2M2-Net: Part-Aware Prompt-Guided Multimodal Point Cloud Completion

• URL: http://arxiv.org/abs/2312.17611v1
• Date: Fri, 29 Dec 2023 14:11:45 GMT
• Title: P2M2-Net: Part-Aware Prompt-Guided Multimodal Point Cloud Completion
• Authors: Linlian Jiang, Pan Chen, Ye Wang, Tieru Wu, Rui Ma
• Abstract summary: Inferring missing regions from severely occluded point clouds is highly challenging. We propose a novel prompt-guided point cloud completion framework, coined P2M2-Net. Given an input partial point cloud and a text prompt describing the part-aware information, our Transformer-based completion network can efficiently fuse the multimodal features.
• Score: 6.407066306127476
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inferring missing regions from severely occluded point clouds is highly challenging. Especially for 3D shapes with rich geometry and structure details, inherent ambiguities of the unknown parts are existing. Existing approaches either learn a one-to-one mapping in a supervised manner or train a generative model to synthesize the missing points for the completion of 3D point cloud shapes. These methods, however, lack the controllability for the completion process and the results are either deterministic or exhibiting uncontrolled diversity. Inspired by the prompt-driven data generation and editing, we propose a novel prompt-guided point cloud completion framework, coined P2M2-Net, to enable more controllable and more diverse shape completion. Given an input partial point cloud and a text prompt describing the part-aware information such as semantics and structure of the missing region, our Transformer-based completion network can efficiently fuse the multimodal features and generate diverse results following the prompt guidance. We train the P2M2-Net on a new large-scale PartNet-Prompt dataset and conduct extensive experiments on two challenging shape completion benchmarks. Quantitative and qualitative results show the efficacy of incorporating prompts for more controllable part-aware point cloud completion and generation. Code and data are available at https://github.com/JLU-ICL/P2M2-Net.

Related papers

• DuInNet: Dual-Modality Feature Interaction for Point Cloud Completion [33.023714580710504]
  We contribute a large-scale multimodal point cloud completion benchmark ModelNet-MPC with richer shape categories and more diverse test data. Besides the fully supervised point cloud completion task, two additional tasks including denoising completion and zero-shot learning completion are proposed. Experiments on the ShapeNet-ViPC and ModelNet-MPC benchmarks demonstrate that DuInNet exhibits superiority, robustness and transfer ability in all completion tasks over state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-10T05:19:40Z)
• Rotation-Invariant Completion Network [8.023732679237021]
  Real-world point clouds usually suffer from incompleteness and display different poses. Current point cloud completion methods excel in reproducing complete point clouds with consistent poses as seen in the training set. We propose a network named Rotation-Invariant Completion Network (RICNet), which consists of two parts: a Dual Pipeline Completion Network (DPCNet) and an enhancing module.
  arXiv  Detail & Related papers  (2023-08-23T07:58:20Z)
• DualGenerator: Information Interaction-based Generative Network for Point Cloud Completion [25.194587599472147]
  Point cloud completion estimates complete shapes from incomplete point clouds to obtain higher-quality point cloud data. Most existing methods only consider global object features, ignoring spatial and semantic information of adjacent points. We propose an information interaction-based generative network for point cloud completion.
  arXiv  Detail & Related papers  (2023-05-16T03:25:38Z)
• Variational Relational Point Completion Network for Robust 3D Classification [59.80993960827833]
  Vari point cloud completion methods tend to generate global shape skeletons hence lack fine local details. This paper proposes a variational framework, point Completion Network (VRCNet) with two appealing properties. VRCNet shows great generalizability and robustness on real-world point cloud scans.
  arXiv  Detail & Related papers  (2023-04-18T17:03:20Z)
• Point2Vec for Self-Supervised Representation Learning on Point Clouds [66.53955515020053]
  We extend data2vec to the point cloud domain and report encouraging results on several downstream tasks. We propose point2vec, which unleashes the full potential of data2vec-like pre-training on point clouds.
  arXiv  Detail & Related papers  (2023-03-29T10:08:29Z)
• CompleteDT: Point Cloud Completion with Dense Augment Inference Transformers [14.823742295692856]
  Point cloud completion task aims to predict the missing part of incomplete point clouds and generate point clouds with details. We propose a novel point cloud completion network, CompleteDT, which is based on the transformer.
  arXiv  Detail & Related papers  (2022-05-30T11:17:31Z)
• Point cloud completion on structured feature map with feedback network [28.710494879042002]
  We propose FSNet, a feature structuring module that can adaptively aggregate point-wise features into a 2D structured feature map. A 2D convolutional neural network is adopted to decode feature maps from FSNet into a coarse and complete point cloud. A point cloud upsampling network is used to generate dense point cloud from the partial input and the coarse intermediate output.
  arXiv  Detail & Related papers  (2022-02-17T10:59:40Z)
• A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion [69.32451612060214]
  Real-scanned 3D point clouds are often incomplete, and it is important to recover complete point clouds for downstream applications. Most existing point cloud completion methods use Chamfer Distance (CD) loss for training. We propose a novel Point Diffusion-Refinement (PDR) paradigm for point cloud completion.
  arXiv  Detail & Related papers  (2021-12-07T06:59:06Z)
• PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers [81.71904691925428]
  We present a new method that reformulates point cloud completion as a set-to-set translation problem. We also design a new model, called PoinTr, that adopts a transformer encoder-decoder architecture for point cloud completion. Our method outperforms state-of-the-art methods by a large margin on both the new benchmarks and the existing ones.
  arXiv  Detail & Related papers  (2021-08-19T17:58:56Z)
• Cascaded Refinement Network for Point Cloud Completion with Self-supervision [74.80746431691938]
  We introduce a two-branch network for shape completion. The first branch is a cascaded shape completion sub-network to synthesize complete objects. The second branch is an auto-encoder to reconstruct the original partial input.
  arXiv  Detail & Related papers  (2020-10-17T04:56:22Z)
• GRNet: Gridding Residual Network for Dense Point Cloud Completion [54.43648460932248]
  Estimating the complete 3D point cloud from an incomplete one is a key problem in many vision and robotics applications. We propose a novel Gridding Residual Network (GRNet) for point cloud completion. Experimental results indicate that the proposed GRNet performs favorably against state-of-the-art methods on the ShapeNet, Completion3D, and KITTI benchmarks.
  arXiv  Detail & Related papers  (2020-06-06T02:46:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.