CCD-3DR: Consistent Conditioning in Diffusion for Single-Image 3D Reconstruction

• URL: http://arxiv.org/abs/2308.07837v1
• Date: Tue, 15 Aug 2023 15:27:42 GMT
• Title: CCD-3DR: Consistent Conditioning in Diffusion for Single-Image 3D Reconstruction
• Authors: Yan Di, Chenyangguang Zhang, Pengyuan Wang, Guangyao Zhai, Ruida Zhang, Fabian Manhardt, Benjamin Busam, Xiangyang Ji, and Federico Tombari
• Abstract summary: We present CCD-3DR, which exploits a novel centered diffusion probabilistic model for consistent local feature conditioning. CCD-3DR outperforms all competitors by a large margin, with over 40% improvement.
• Score: 81.98244738773766
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we present a novel shape reconstruction method leveraging diffusion model to generate 3D sparse point cloud for the object captured in a single RGB image. Recent methods typically leverage global embedding or local projection-based features as the condition to guide the diffusion model. However, such strategies fail to consistently align the denoised point cloud with the given image, leading to unstable conditioning and inferior performance. In this paper, we present CCD-3DR, which exploits a novel centered diffusion probabilistic model for consistent local feature conditioning. We constrain the noise and sampled point cloud from the diffusion model into a subspace where the point cloud center remains unchanged during the forward diffusion process and reverse process. The stable point cloud center further serves as an anchor to align each point with its corresponding local projection-based features. Extensive experiments on synthetic benchmark ShapeNet-R2N2 demonstrate that CCD-3DR outperforms all competitors by a large margin, with over 40% improvement. We also provide results on real-world dataset Pix3D to thoroughly demonstrate the potential of CCD-3DR in real-world applications. Codes will be released soon

Related papers

• Diffusion-Occ: 3D Point Cloud Completion via Occupancy Diffusion [5.189790379672664]
  We introduce textbfDiffusion-Occ, a novel framework for Diffusion Point Cloud Completion. By thresholding the occupancy field, we convert it into a complete point cloud. Experimental results demonstrate that Diffusion-Occ outperforms existing discriminative and generative methods.
  arXiv  Detail & Related papers  (2024-08-27T07:57:58Z)
• Uplifting Range-View-based 3D Semantic Segmentation in Real-Time with Multi-Sensor Fusion [18.431017678057348]
  Range-View(RV)-based 3D point cloud segmentation is widely adopted due to its compact data form. However, RV-based methods fall short in providing robust segmentation for the occluded points. We propose a new LiDAR and Camera Range-view-based 3D point cloud semantic segmentation method (LaCRange) In addition to being real-time, the proposed method achieves state-of-the-art results on nuScenes benchmark.
  arXiv  Detail & Related papers  (2024-07-12T21:41:57Z)
• Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation [29.818827785812086]
  Controllable generation of 3D assets is important for many practical applications like content creation in movies, games and engineering, as well as in AR/VR. We present a suitable representation for 3D diffusion models to enable disentanglement by introducing a hybrid point cloud and neural radiance field approach.
  arXiv  Detail & Related papers  (2023-12-21T18:46:27Z)
• D-SCo: Dual-Stream Conditional Diffusion for Monocular Hand-Held Object Reconstruction [74.49121940466675]
  We introduce centroid-fixed dual-stream conditional diffusion for monocular hand-held object reconstruction. First, to avoid the object centroid from deviating, we utilize a novel hand-constrained centroid fixing paradigm. Second, we introduce a dual-stream denoiser to semantically and geometrically model hand-object interactions.
  arXiv  Detail & Related papers  (2023-11-23T20:14:50Z)
• StarNet: Style-Aware 3D Point Cloud Generation [82.30389817015877]
  StarNet is able to reconstruct and generate high-fidelity and even 3D point clouds using a mapping network. Our framework achieves comparable state-of-the-art performance on various metrics in the point cloud reconstruction and generation tasks.
  arXiv  Detail & Related papers  (2023-03-28T08:21:44Z)
• $PC^2$: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction [97.06927852165464]
  Reconstructing the 3D shape of an object from a single RGB image is a long-standing and highly challenging problem in computer vision. We propose a novel method for single-image 3D reconstruction which generates a sparse point cloud via a conditional denoising diffusion process.
  arXiv  Detail & Related papers  (2023-02-21T13:37:07Z)
• A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion [69.32451612060214]
  Real-scanned 3D point clouds are often incomplete, and it is important to recover complete point clouds for downstream applications. Most existing point cloud completion methods use Chamfer Distance (CD) loss for training. We propose a novel Point Diffusion-Refinement (PDR) paradigm for point cloud completion.
  arXiv  Detail & Related papers  (2021-12-07T06:59:06Z)
• Pseudo-LiDAR Point Cloud Interpolation Based on 3D Motion Representation and Spatial Supervision [68.35777836993212]
  We propose a Pseudo-LiDAR point cloud network to generate temporally and spatially high-quality point cloud sequences. By exploiting the scene flow between point clouds, the proposed network is able to learn a more accurate representation of the 3D spatial motion relationship.
  arXiv  Detail & Related papers  (2020-06-20T03:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.