DiffComplete: Diffusion-based Generative 3D Shape Completion
- URL: http://arxiv.org/abs/2306.16329v1
- Date: Wed, 28 Jun 2023 16:07:36 GMT
- Title: DiffComplete: Diffusion-based Generative 3D Shape Completion
- Authors: Ruihang Chu, Enze Xie, Shentong Mo, Zhenguo Li, Matthias Nie{\ss}ner,
Chi-Wing Fu, Jiaya Jia
- Abstract summary: We introduce a new diffusion-based approach for shape completion on 3D range scans.
We strike a balance between realism, multi-modality, and high fidelity.
DiffComplete sets a new SOTA performance on two large-scale 3D shape completion benchmarks.
- Score: 114.43353365917015
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce a new diffusion-based approach for shape completion on 3D range
scans. Compared with prior deterministic and probabilistic methods, we strike a
balance between realism, multi-modality, and high fidelity. We propose
DiffComplete by casting shape completion as a generative task conditioned on
the incomplete shape. Our key designs are two-fold. First, we devise a
hierarchical feature aggregation mechanism to inject conditional features in a
spatially-consistent manner. So, we can capture both local details and broader
contexts of the conditional inputs to control the shape completion. Second, we
propose an occupancy-aware fusion strategy in our model to enable the
completion of multiple partial shapes and introduce higher flexibility on the
input conditions. DiffComplete sets a new SOTA performance (e.g., 40% decrease
on l_1 error) on two large-scale 3D shape completion benchmarks. Our completed
shapes not only have a realistic outlook compared with the deterministic
methods but also exhibit high similarity to the ground truths compared with the
probabilistic alternatives. Further, DiffComplete has strong generalizability
on objects of entirely unseen classes for both synthetic and real data,
eliminating the need for model re-training in various applications.
Related papers
- RealDiff: Real-world 3D Shape Completion using Self-Supervised Diffusion Models [15.209079637302905]
We propose a self-supervised framework, namely RealDiff, that formulates point cloud completion as a conditional generation problem directly on real-world measurements.
Specifically, RealDiff simulates a diffusion process at the missing object parts while conditioning the generation on the partial input to address the multimodal nature of the task.
Experimental results show that our method consistently outperforms state-of-the-art methods in real-world point cloud completion.
arXiv Detail & Related papers (2024-09-16T11:18:57Z) - SC-Diff: 3D Shape Completion with Latent Diffusion Models [4.913210912019975]
This paper introduces a 3D shape completion approach using a 3D latent diffusion model optimized for completing shapes.
Our method combines image-based conditioning through cross-attention and spatial conditioning through the integration of 3D features from captured partial scans.
arXiv Detail & Related papers (2024-03-19T06:01:11Z) - Robust 3D Tracking with Quality-Aware Shape Completion [67.9748164949519]
We propose a synthetic target representation composed of dense and complete point clouds depicting the target shape precisely by shape completion for robust 3D tracking.
Specifically, we design a voxelized 3D tracking framework with shape completion, in which we propose a quality-aware shape completion mechanism to alleviate the adverse effect of noisy historical predictions.
arXiv Detail & Related papers (2023-12-17T04:50:24Z) - T1: Scaling Diffusion Probabilistic Fields to High-Resolution on Unified
Visual Modalities [69.16656086708291]
Diffusion Probabilistic Field (DPF) models the distribution of continuous functions defined over metric spaces.
We propose a new model comprising of a view-wise sampling algorithm to focus on local structure learning.
The model can be scaled to generate high-resolution data while unifying multiple modalities.
arXiv Detail & Related papers (2023-05-24T03:32:03Z) - PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval
and Deformation [59.70430570779819]
We introduce a data-driven shape completion approach that focuses on completing geometric details of missing regions of 3D shapes.
Our key insight is to copy and deform patches from the partial input to complete missing regions.
We leverage repeating patterns by retrieving patches from the partial input, and learn global structural priors by using a neural network to guide the retrieval and deformation steps.
arXiv Detail & Related papers (2022-07-24T18:59:09Z) - Probabilistic Implicit Scene Completion [6.954686339092988]
We propose a probabilistic shape completion method extended to the continuous geometry of large-scale 3D scenes.
We employ the Generative Cellular Automata that learns the multi-modal distribution and transform the formulation to process large-scale continuous geometry.
Experiments show that our model successfully generates diverse plausible scenes faithful to the input, especially when the input suffers from a significant amount of missing data.
arXiv Detail & Related papers (2022-04-04T06:16:54Z) - Probabilistic Registration for Gaussian Process 3D shape modelling in
the presence of extensive missing data [63.8376359764052]
We propose a shape fitting/registration method based on a Gaussian Processes formulation, suitable for shapes with extensive regions of missing data.
Experiments are conducted both for a 2D small dataset with diverse transformations and a 3D dataset of ears.
arXiv Detail & Related papers (2022-03-26T16:48:27Z) - AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation [29.018733252938926]
Powerful priors allow us to perform inference with insufficient information.
We propose an autoregressive prior for 3D shapes to solve multimodal 3D tasks such as shape completion, reconstruction, and generation.
arXiv Detail & Related papers (2022-03-17T17:59:54Z) - ShapeFormer: Transformer-based Shape Completion via Sparse
Representation [41.33457875133559]
We present ShapeFormer, a network that produces a distribution of object completions conditioned on incomplete, and possibly noisy, point clouds.
The resultant distribution can then be sampled to generate likely completions, each exhibiting plausible shape details while being faithful to the input.
arXiv Detail & Related papers (2022-01-25T13:58:30Z) - Unsupervised 3D Shape Completion through GAN Inversion [116.27680045885849]
We present ShapeInversion, which introduces Generative Adrial Network (GAN) inversion to shape completion for the first time.
ShapeInversion uses a GAN pre-trained on complete shapes by searching for a latent code that gives a complete shape that best fits the given partial input.
On the ShapeNet benchmark, the proposed ShapeInversion outperforms the SOTA unsupervised method, and is comparable with supervised methods that are learned using paired data.
arXiv Detail & Related papers (2021-04-27T17:53:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.