Multiscale Tensor Decomposition and Rendering Equation Encoding for View Synthesis

• URL: http://arxiv.org/abs/2303.03808v2
• Date: Sat, 27 May 2023 07:21:49 GMT
• Title: Multiscale Tensor Decomposition and Rendering Equation Encoding for View Synthesis
• Authors: Kang Han, Wei Xiang
• Abstract summary: We propose a novel approach dubbed the neural radiance feature field (NRFF) NRFF improves state-of-the-art rendering results by over 1 dB in PSNR on both the NeRF and NSVF datasets.
• Score: 7.680742911100444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rendering novel views from captured multi-view images has made considerable progress since the emergence of the neural radiance field. This paper aims to further advance the quality of view synthesis by proposing a novel approach dubbed the neural radiance feature field (NRFF). We first propose a multiscale tensor decomposition scheme to organize learnable features so as to represent scenes from coarse to fine scales. We demonstrate many benefits of the proposed multiscale representation, including more accurate scene shape and appearance reconstruction, and faster convergence compared with the single-scale representation. Instead of encoding view directions to model view-dependent effects, we further propose to encode the rendering equation in the feature space by employing the anisotropic spherical Gaussian mixture predicted from the proposed multiscale representation. The proposed NRFF improves state-of-the-art rendering results by over 1 dB in PSNR on both the NeRF and NSVF synthetic datasets. A significant improvement has also been observed on the real-world Tanks & Temples dataset. Code can be found at https://github.com/imkanghan/nrff.

Related papers

• Splatfacto-W: A Nerfstudio Implementation of Gaussian Splatting for Unconstrained Photo Collections [25.154665328053333]
  We introduce Splatfacto-W, an in-trivial approach that integrates per-Gaussian neural color features and per-image appearance embeddings into an rendering process. Our method improves the Peak Signal-to-Noise Ratio (PSNR) by an average of 5.3 dB compared to 3DGS, enhances training speed by 150 times compared to NeRF-based methods, and achieves a similar rendering speed to 3DGS.
  arXiv  Detail & Related papers  (2024-07-17T04:02:54Z)
• PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields [54.8553158441296]
  We propose a novel visual localization framework, ie, PNeRFLoc, based on a unified point-based representation. On the one hand, PNeRFLoc supports the initial pose estimation by matching 2D and 3D feature points. On the other hand, it also enables pose refinement with novel view synthesis using rendering-based optimization.
  arXiv  Detail & Related papers  (2023-12-17T08:30:00Z)
• Anisotropic Neural Representation Learning for High-Quality Neural Rendering [0.0]
  We propose an anisotropic neural representation learning method that utilizes learnable view-dependent features to improve scene representation and reconstruction. Our method is flexiable and can be plugged into NeRF-based frameworks.
  arXiv  Detail & Related papers  (2023-11-30T07:29:30Z)
• Anti-Aliased Neural Implicit Surfaces with Encoding Level of Detail [54.03399077258403]
  We present LoD-NeuS, an efficient neural representation for high-frequency geometry detail recovery and anti-aliased novel view rendering. Our representation aggregates space features from a multi-convolved featurization within a conical frustum along a ray.
  arXiv  Detail & Related papers  (2023-09-19T05:44:00Z)
• Learning Neural Duplex Radiance Fields for Real-Time View Synthesis [33.54507228895688]
  We propose a novel approach to distill and bake NeRFs into highly efficient mesh-based neural representations. We demonstrate the effectiveness and superiority of our approach via extensive experiments on a range of standard datasets.
  arXiv  Detail & Related papers  (2023-04-20T17:59:52Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• Cascaded and Generalizable Neural Radiance Fields for Fast View Synthesis [35.035125537722514]
  We present CG-NeRF, a cascade and generalizable neural radiance fields method for view synthesis. We first train CG-NeRF on multiple 3D scenes of the DTU dataset. We show that CG-NeRF outperforms state-of-the-art generalizable neural rendering methods on various synthetic and real datasets.
  arXiv  Detail & Related papers  (2022-08-09T12:23:48Z)
• InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering [55.70938412352287]
  We present an information-theoretic regularization technique for few-shot novel view synthesis based on neural implicit representation. The proposed approach minimizes potential reconstruction inconsistency that happens due to insufficient viewpoints. We achieve consistently improved performance compared to existing neural view synthesis methods by large margins on multiple standard benchmarks.
  arXiv  Detail & Related papers  (2021-12-31T11:56:01Z)
• NeRF in detail: Learning to sample for view synthesis [104.75126790300735]
  Neural radiance fields (NeRF) methods have demonstrated impressive novel view synthesis. In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a performance and not trained end-to-end for the task at hand. We introduce a differentiable module that learns to propose samples and their importance for the fine network, and consider and compare multiple alternatives for its neural architecture.
  arXiv  Detail & Related papers  (2021-06-09T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.