View Synthesis with Sculpted Neural Points
- URL: http://arxiv.org/abs/2205.05869v1
- Date: Thu, 12 May 2022 03:54:35 GMT
- Title: View Synthesis with Sculpted Neural Points
- Authors: Yiming Zuo, Jia Deng
- Abstract summary: Implicit neural representations have achieved impressive visual quality but have drawbacks in computational efficiency.
We propose a new approach that performs view synthesis using point clouds.
It is the first point-based method to achieve better visual quality than NeRF while being more than 100x faster in rendering speed.
- Score: 64.40344086212279
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We address the task of view synthesis, which can be posed as recovering a
rendering function that renders new views from a set of existing images. In
many recent works such as NeRF, this rendering function is parameterized using
implicit neural representations of scene geometry. Implicit neural
representations have achieved impressive visual quality but have drawbacks in
computational efficiency. In this work, we propose a new approach that performs
view synthesis using point clouds. It is the first point-based method to
achieve better visual quality than NeRF while being more than 100x faster in
rendering speed. Our approach builds on existing works on differentiable
point-based rendering but introduces a novel technique we call "Sculpted Neural
Points (SNP)", which significantly improves the robustness to errors and holes
in the reconstructed point cloud. Experiments show that on the task of view
synthesis, our sculpting technique closes the gap between point-based and
implicit representation-based methods. Code is available at
https://github.com/princeton-vl/SNP and supplementary video at
https://youtu.be/dBwCQP9uNws.
Related papers
- MomentsNeRF: Leveraging Orthogonal Moments for Few-Shot Neural Rendering [4.6786468967610055]
We propose MomentsNeRF, a novel framework for one- and few-shot neural rendering.
Our architecture offers a new transfer learning method to train on multi-scenes.
Our approach is the first to successfully harness features extracted from Gabor and Zernike moments.
arXiv Detail & Related papers (2024-07-02T21:02:48Z) - D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video [53.83936023443193]
This paper contributes to the field by introducing a new synthesis method for dynamic novel view from monocular video, such as smartphone captures.
Our approach represents the as a $textitdynamic neural point cloud$, an implicit time-conditioned point cloud that encodes local geometry and appearance in separate hash-encoded neural feature grids.
arXiv Detail & Related papers (2024-06-14T14:35:44Z) - PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields [54.8553158441296]
We propose a novel visual localization framework, ie, PNeRFLoc, based on a unified point-based representation.
On the one hand, PNeRFLoc supports the initial pose estimation by matching 2D and 3D feature points.
On the other hand, it also enables pose refinement with novel view synthesis using rendering-based optimization.
arXiv Detail & Related papers (2023-12-17T08:30:00Z) - Re-Nerfing: Improving Novel Views Synthesis through Novel Views Synthesis [80.3686833921072]
Re-Nerfing is a simple and general multi-stage data augmentation approach.
We train a NeRF with the available views, then use the optimized NeRF to synthesize pseudo-views around the original ones.
We also train a second NeRF with both the original images and the pseudo views masking out uncertain regions.
arXiv Detail & Related papers (2023-12-04T18:56:08Z) - HQ3DAvatar: High Quality Controllable 3D Head Avatar [65.70885416855782]
This paper presents a novel approach to building highly photorealistic digital head avatars.
Our method learns a canonical space via an implicit function parameterized by a neural network.
At test time, our method is driven by a monocular RGB video.
arXiv Detail & Related papers (2023-03-25T13:56:33Z) - NPBG++: Accelerating Neural Point-Based Graphics [14.366073496519139]
NPBG++ is a novel view synthesis (NVS) task that achieves high rendering realism with low scene fitting time.
Our method efficiently leverages the multiview observations and the point cloud of a static scene to predict a neural descriptor for each point.
In our comparisons, the proposed system outperforms previous NVS approaches in terms of fitting and rendering runtimes while producing images of similar quality.
arXiv Detail & Related papers (2022-03-24T19:59:39Z) - InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering [55.70938412352287]
We present an information-theoretic regularization technique for few-shot novel view synthesis based on neural implicit representation.
The proposed approach minimizes potential reconstruction inconsistency that happens due to insufficient viewpoints.
We achieve consistently improved performance compared to existing neural view synthesis methods by large margins on multiple standard benchmarks.
arXiv Detail & Related papers (2021-12-31T11:56:01Z) - pixelNeRF: Neural Radiance Fields from One or Few Images [20.607712035278315]
pixelNeRF is a learning framework that predicts a continuous neural scene representation conditioned on one or few input images.
We conduct experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects.
In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction.
arXiv Detail & Related papers (2020-12-03T18:59:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.