Point2Pix: Photo-Realistic Point Cloud Rendering via Neural Radiance Fields

• URL: http://arxiv.org/abs/2303.16482v1
• Date: Wed, 29 Mar 2023 06:26:55 GMT
• Title: Point2Pix: Photo-Realistic Point Cloud Rendering via Neural Radiance Fields
• Authors: Tao Hu, Xiaogang Xu, Shu Liu, Jiaya Jia
• Abstract summary: Recent Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. We present Point2Pix as a novel point to link the 3D sparse point clouds with 2D dense image pixels.
• Score: 63.21420081888606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthesizing photo-realistic images from a point cloud is challenging because of the sparsity of point cloud representation. Recent Neural Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. In this paper, we present Point2Pix as a novel point renderer to link the 3D sparse point clouds with 2D dense image pixels. Taking advantage of the point cloud 3D prior and NeRF rendering pipeline, our method can synthesize high-quality images from colored point clouds, generally for novel indoor scenes. To improve the efficiency of ray sampling, we propose point-guided sampling, which focuses on valid samples. Also, we present Point Encoding to build Multi-scale Radiance Fields that provide discriminative 3D point features. Finally, we propose Fusion Encoding to efficiently synthesize high-quality images. Extensive experiments on the ScanNet and ArkitScenes datasets demonstrate the effectiveness and generalization.

Related papers

• GaussianPU: A Hybrid 2D-3D Upsampling Framework for Enhancing Color Point Clouds via 3D Gaussian Splatting [11.60605616190011]
  We propose a novel 2D-3D hybrid colored point cloud upsampling framework (GaussianPU) based on 3D Gaussian Splatting (3DGS) for robotic perception. A dual scale rendered image restoration network transforms sparse point cloud renderings into dense representations. We have made a series of enhancements to the vanilla 3DGS, enabling precise control over the number of points.
  arXiv  Detail & Related papers  (2024-09-03T03:35:04Z)
• TriVol: Point Cloud Rendering via Triple Volumes [57.305748806545026]
  We present a dense while lightweight 3D representation, named TriVol, that can be combined with NeRF to render photo-realistic images from point clouds. Our framework has excellent generalization ability to render a category of scenes/objects without fine-tuning.
  arXiv  Detail & Related papers  (2023-03-29T06:34:12Z)
• Ponder: Point Cloud Pre-training via Neural Rendering [93.34522605321514]
  We propose a novel approach to self-supervised learning of point cloud representations by differentiable neural encoders. The learned point-cloud can be easily integrated into various downstream tasks, including not only high-level rendering tasks like 3D detection and segmentation, but low-level tasks like 3D reconstruction and image rendering.
  arXiv  Detail & Related papers  (2022-12-31T08:58:39Z)
• Points2NeRF: Generating Neural Radiance Fields from 3D point cloud [0.0]
  We propose representing 3D objects as Neural Radiance Fields (NeRFs) We leverage a hypernetwork paradigm and train the model to take a 3D point cloud with the associated color values. Our method provides efficient 3D object representation and offers several advantages over the existing approaches.
  arXiv  Detail & Related papers  (2022-06-02T20:23:33Z)
• VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusion [62.24001258298076]
  VPFNet is a new architecture that cleverly aligns and aggregates the point cloud and image data at the virtual' points. Our VPFNet achieves 83.21% moderate 3D AP and 91.86% moderate BEV AP on the KITTI test set, ranking the 1st since May 21th, 2021.
  arXiv  Detail & Related papers  (2021-11-29T08:51:20Z)
• Graphite: GRAPH-Induced feaTure Extraction for Point Cloud Registration [80.69255347486693]
  We introduce a GRAPH-Induced feaTure Extraction pipeline, a simple yet powerful feature and keypoint detector. We construct a generic graph-based learning scheme to describe point cloud regions and extract salient points. We Reformulate the 3D keypoint pipeline with graph neural networks which allow efficient processing of the point set.
  arXiv  Detail & Related papers  (2020-10-18T19:41:09Z)
• ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes [93.82668222075128]
  We propose a 3D detection architecture called ImVoteNet for RGB-D scenes. ImVoteNet is based on fusing 2D votes in images and 3D votes in point clouds. We validate our model on the challenging SUN RGB-D dataset.
  arXiv  Detail & Related papers  (2020-01-29T05:09:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.