PointNeRF++: A multi-scale, point-based Neural Radiance Field

• URL: http://arxiv.org/abs/2312.02362v2
• Date: Thu, 21 Mar 2024 21:28:37 GMT
• Title: PointNeRF++: A multi-scale, point-based Neural Radiance Field
• Authors: Weiwei Sun, Eduard Trulls, Yang-Che Tseng, Sneha Sambandam, Gopal Sharma, Andrea Tagliasacchi, Kwang Moo Yi,
• Abstract summary: Point clouds offer an attractive source of information to complement images in neural scene representations. NeRF rendering methods based on point clouds do not perform well when the point cloud quality is low. We present a representation that aggregates point clouds at multiple scale levels with sparse voxel grids at different resolutions. We validate our method on the NeRF Synthetic, ScanNet, and KITTI-360 datasets, outperforming the state of the art.
• Score: 31.23973383531481
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Point clouds offer an attractive source of information to complement images in neural scene representations, especially when few images are available. Neural rendering methods based on point clouds do exist, but they do not perform well when the point cloud quality is low -- e.g., sparse or incomplete, which is often the case with real-world data. We overcome these problems with a simple representation that aggregates point clouds at multiple scale levels with sparse voxel grids at different resolutions. To deal with point cloud sparsity, we average across multiple scale levels -- but only among those that are valid, i.e., that have enough neighboring points in proximity to the ray of a pixel. To help model areas without points, we add a global voxel at the coarsest scale, thus unifying ``classical'' and point-based NeRF formulations. We validate our method on the NeRF Synthetic, ScanNet, and KITTI-360 datasets, outperforming the state of the art, with a significant gap compared to other NeRF-based methods, especially on more challenging scenes.

Related papers

• Point Cloud Compression with Implicit Neural Representations: A Unified Framework [54.119415852585306]
  We present a pioneering point cloud compression framework capable of handling both geometry and attribute components. Our framework utilizes two coordinate-based neural networks to implicitly represent a voxelized point cloud. Our method exhibits high universality when contrasted with existing learning-based techniques.
  arXiv  Detail & Related papers  (2024-05-19T09:19:40Z)
• GPN: Generative Point-based NeRF [0.65268245109828]
  We propose using Generative Point-based NeRF (GPN) to reconstruct and repair a partial cloud. The repaired point cloud can achieve multi-view consistency with the captured images at high spatial resolution.
  arXiv  Detail & Related papers  (2024-04-12T08:14:17Z)
• Point2Pix: Photo-Realistic Point Cloud Rendering via Neural Radiance Fields [63.21420081888606]
  Recent Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. We present Point2Pix as a novel point to link the 3D sparse point clouds with 2D dense image pixels.
  arXiv  Detail & Related papers  (2023-03-29T06:26:55Z)
• Ponder: Point Cloud Pre-training via Neural Rendering [93.34522605321514]
  We propose a novel approach to self-supervised learning of point cloud representations by differentiable neural encoders. The learned point-cloud can be easily integrated into various downstream tasks, including not only high-level rendering tasks like 3D detection and segmentation, but low-level tasks like 3D reconstruction and image rendering.
  arXiv  Detail & Related papers  (2022-12-31T08:58:39Z)
• Boosting Point Clouds Rendering via Radiance Mapping [49.24193509772339]
  We focus on boosting the image quality of point clouds rendering with a compact model design. We simplify the NeRF representation to a spatial mapping function which only requires single evaluation per pixel. Our method achieves the state-of-the-art rendering on point clouds, outperforming prior works by notable margins.
  arXiv  Detail & Related papers  (2022-10-27T01:25:57Z)
• Shrinking unit: a Graph Convolution-Based Unit for CNN-like 3D Point Cloud Feature Extractors [0.0]
  We argue that a lack of inspiration from the image domain might be the primary cause of such a gap. We propose a graph convolution-based unit, dubbed Shrinking unit, that can be stacked vertically and horizontally for the design of CNN-like 3D point cloud feature extractors.
  arXiv  Detail & Related papers  (2022-09-26T15:28:31Z)
• PointAttN: You Only Need Attention for Point Cloud Completion [89.88766317412052]
  Point cloud completion refers to completing 3D shapes from partial 3D point clouds. We propose a novel neural network for processing point cloud in a per-point manner to eliminate kNNs. The proposed framework, namely PointAttN, is simple, neat and effective, which can precisely capture the structural information of 3D shapes.
  arXiv  Detail & Related papers  (2022-03-16T09:20:01Z)
• Learning Neural Light Fields with Ray-Space Embedding Networks [51.88457861982689]
  We propose a novel neural light field representation that is compact and directly predicts integrated radiance along rays. Our method achieves state-of-the-art quality on dense forward-facing datasets such as the Stanford Light Field dataset.
  arXiv  Detail & Related papers  (2021-12-02T18:59:51Z)
• SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering [21.563862632172363]
  We propose a self-supervised point cloud upsampling network (SSPU-Net) to generate dense point clouds without using ground truth. To achieve this, we exploit the consistency between the input sparse point cloud and generated dense point cloud for the shapes and rendered images.
  arXiv  Detail & Related papers  (2021-08-01T13:26:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.