GPN: Generative Point-based NeRF

• URL: http://arxiv.org/abs/2404.08312v1
• Date: Fri, 12 Apr 2024 08:14:17 GMT
• Title: GPN: Generative Point-based NeRF
• Authors: Haipeng Wang,
• Abstract summary: We propose using Generative Point-based NeRF (GPN) to reconstruct and repair a partial cloud. The repaired point cloud can achieve multi-view consistency with the captured images at high spatial resolution.
• Score: 0.65268245109828
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scanning real-life scenes with modern registration devices typically gives incomplete point cloud representations, primarily due to the limitations of partial scanning, 3D occlusions, and dynamic light conditions. Recent works on processing incomplete point clouds have always focused on point cloud completion. However, these approaches do not ensure consistency between the completed point cloud and the captured images regarding color and geometry. We propose using Generative Point-based NeRF (GPN) to reconstruct and repair a partial cloud by fully utilizing the scanning images and the corresponding reconstructed cloud. The repaired point cloud can achieve multi-view consistency with the captured images at high spatial resolution. For the finetunes of a single scene, we optimize the global latent condition by incorporating an Auto-Decoder architecture while retaining multi-view consistency. As a result, the generated point clouds are smooth, plausible, and geometrically consistent with the partial scanning images. Extensive experiments on ShapeNet demonstrate that our works achieve competitive performances to the other state-of-the-art point cloud-based neural scene rendering and editing performances.

Related papers

• Zero-shot Point Cloud Completion Via 2D Priors [52.72867922938023]
  3D point cloud completion is designed to recover complete shapes from partially observed point clouds. We propose a zero-shot framework aimed at completing partially observed point clouds across any unseen categories.
  arXiv  Detail & Related papers  (2024-04-10T08:02:17Z)
• PointNeRF++: A multi-scale, point-based Neural Radiance Field [31.23973383531481]
  Point clouds offer an attractive source of information to complement images in neural scene representations. NeRF rendering methods based on point clouds do not perform well when the point cloud quality is low. We present a representation that aggregates point clouds at multiple scale levels with sparse voxel grids at different resolutions. We validate our method on the NeRF Synthetic, ScanNet, and KITTI-360 datasets, outperforming the state of the art.
  arXiv  Detail & Related papers  (2023-12-04T21:43:00Z)
• Point2Pix: Photo-Realistic Point Cloud Rendering via Neural Radiance Fields [63.21420081888606]
  Recent Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. We present Point2Pix as a novel point to link the 3D sparse point clouds with 2D dense image pixels.
  arXiv  Detail & Related papers  (2023-03-29T06:26:55Z)
• Ponder: Point Cloud Pre-training via Neural Rendering [93.34522605321514]
  We propose a novel approach to self-supervised learning of point cloud representations by differentiable neural encoders. The learned point-cloud can be easily integrated into various downstream tasks, including not only high-level rendering tasks like 3D detection and segmentation, but low-level tasks like 3D reconstruction and image rendering.
  arXiv  Detail & Related papers  (2022-12-31T08:58:39Z)
• Leveraging Single-View Images for Unsupervised 3D Point Cloud Completion [53.93172686610741]
  Cross-PCC is an unsupervised point cloud completion method without requiring any 3D complete point clouds. To take advantage of the complementary information from 2D images, we use a single-view RGB image to extract 2D features. Our method even achieves comparable performance to some supervised methods.
  arXiv  Detail & Related papers  (2022-12-01T15:11:21Z)
• Boosting Point Clouds Rendering via Radiance Mapping [49.24193509772339]
  We focus on boosting the image quality of point clouds rendering with a compact model design. We simplify the NeRF representation to a spatial mapping function which only requires single evaluation per pixel. Our method achieves the state-of-the-art rendering on point clouds, outperforming prior works by notable margins.
  arXiv  Detail & Related papers  (2022-10-27T01:25:57Z)
• Learning a Structured Latent Space for Unsupervised Point Cloud Completion [48.79411151132766]
  We propose a novel framework, which learns a unified and structured latent space that encoding both partial and complete point clouds. Our proposed method consistently outperforms state-of-the-art unsupervised methods on both synthetic ShapeNet and real-world KITTI, ScanNet, and Matterport3D datasets.
  arXiv  Detail & Related papers  (2022-03-29T13:58:44Z)
• Temporal Point Cloud Completion with Pose Disturbance [32.536545112645335]
  We provide complete point clouds from sparse input with pose disturbance by limited translation and rotation. We also use temporal information to enhance the completion model, refining the output with a sequence of inputs. Our framework is the first to utilize temporal information and ensure temporal consistency with limited transformation.
  arXiv  Detail & Related papers  (2022-02-07T11:41:12Z)
• SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering [21.563862632172363]
  We propose a self-supervised point cloud upsampling network (SSPU-Net) to generate dense point clouds without using ground truth. To achieve this, we exploit the consistency between the input sparse point cloud and generated dense point cloud for the shapes and rendered images.
  arXiv  Detail & Related papers  (2021-08-01T13:26:01Z)
• HyperPocket: Generative Point Cloud Completion [19.895219420937938]
  We introduce a novel autoencoder-based architecture called HyperPocket that disentangles latent representations. We leverage a hypernetwork paradigm to fill the spaces, dubbed pockets, that are left by the missing object parts. Our method offers competitive performances to the other state-of-the-art models.
  arXiv  Detail & Related papers  (2021-02-11T12:30:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.