FPO++: Efficient Encoding and Rendering of Dynamic Neural Radiance Fields by Analyzing and Enhancing Fourier PlenOctrees

• URL: http://arxiv.org/abs/2310.20710v1
• Date: Tue, 31 Oct 2023 17:59:58 GMT
• Title: FPO++: Efficient Encoding and Rendering of Dynamic Neural Radiance Fields by Analyzing and Enhancing Fourier PlenOctrees
• Authors: Saskia Rabich, Patrick Stotko, Reinhard Klein
• Abstract summary: Fourier PlenOctrees have shown to be an efficient representation for real-time rendering of dynamic Neural Radiance Fields (NeRF) In this paper, we perform an in-depth analysis of these artifacts and leverage the resulting insights to propose an improved representation.
• Score: 4.033107207078283
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fourier PlenOctrees have shown to be an efficient representation for real-time rendering of dynamic Neural Radiance Fields (NeRF). Despite its many advantages, this method suffers from artifacts introduced by the involved compression when combining it with recent state-of-the-art techniques for training the static per-frame NeRF models. In this paper, we perform an in-depth analysis of these artifacts and leverage the resulting insights to propose an improved representation. In particular, we present a novel density encoding that adapts the Fourier-based compression to the characteristics of the transfer function used by the underlying volume rendering procedure and leads to a substantial reduction of artifacts in the dynamic model. Furthermore, we show an augmentation of the training data that relaxes the periodicity assumption of the compression. We demonstrate the effectiveness of our enhanced Fourier PlenOctrees in the scope of quantitative and qualitative evaluations on synthetic and real-world scenes.

Related papers

• KFD-NeRF: Rethinking Dynamic NeRF with Kalman Filter [49.85369344101118]
  We introduce KFD-NeRF, a novel dynamic neural radiance field integrated with an efficient and high-quality motion reconstruction framework based on Kalman filtering. Our key idea is to model the dynamic radiance field as a dynamic system whose temporally varying states are estimated based on two sources of knowledge: observations and predictions. Our KFD-NeRF demonstrates similar or even superior performance within comparable computational time and state-of-the-art view synthesis performance with thorough training.
  arXiv  Detail & Related papers  (2024-07-18T05:48:24Z)
• Neural NeRF Compression [19.853882143024]
  Recent NeRFs utilize feature grids to improve rendering quality and speed. These representations introduce significant storage overhead. This paper presents a novel method for efficiently compressing a grid-based NeRF model.
  arXiv  Detail & Related papers  (2024-06-13T09:12:26Z)
• PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields [54.8553158441296]
  We propose a novel visual localization framework, ie, PNeRFLoc, based on a unified point-based representation. On the one hand, PNeRFLoc supports the initial pose estimation by matching 2D and 3D feature points. On the other hand, it also enables pose refinement with novel view synthesis using rendering-based optimization.
  arXiv  Detail & Related papers  (2023-12-17T08:30:00Z)
• Anisotropic Neural Representation Learning for High-Quality Neural Rendering [0.0]
  We propose an anisotropic neural representation learning method that utilizes learnable view-dependent features to improve scene representation and reconstruction. Our method is flexiable and can be plugged into NeRF-based frameworks.
  arXiv  Detail & Related papers  (2023-11-30T07:29:30Z)
• VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations [25.88881764546414]
  VQ-NeRF is an efficient pipeline for enhancing implicit neural representations via vector quantization. We present an innovative multi-scale NeRF sampling scheme that concurrently optimize the NeRF model at both compressed and original scales. We incorporate a semantic loss function to improve the geometric fidelity and semantic coherence of our 3D reconstructions.
  arXiv  Detail & Related papers  (2023-10-23T01:41:38Z)
• Distributed Neural Representation for Reactive in situ Visualization [23.80657290203846]
  Implicit neural representations (INRs) have emerged as a powerful tool for compressing large-scale volume data. We develop a distributed neural representation and optimize it for in situ visualization. Our technique eliminates data exchanges between processes, achieving state-of-the-art compression speed, quality and ratios.
  arXiv  Detail & Related papers  (2023-03-28T03:55:47Z)
• ContraNeRF: Generalizable Neural Radiance Fields for Synthetic-to-real Novel View Synthesis via Contrastive Learning [102.46382882098847]
  We first investigate the effects of synthetic data in synthetic-to-real novel view synthesis. We propose to introduce geometry-aware contrastive learning to learn multi-view consistent features with geometric constraints. Our method can render images with higher quality and better fine-grained details, outperforming existing generalizable novel view synthesis methods in terms of PSNR, SSIM, and LPIPS.
  arXiv  Detail & Related papers  (2023-03-20T12:06:14Z)
• PREF: Phasorial Embedding Fields for Compact Neural Representations [54.44527545923917]
  We present a phasorial embedding field emphPREF as a compact representation to facilitate neural signal modeling and reconstruction tasks. Our experiments show PREF-based neural signal processing technique is on par with the state-of-the-art in 2D image completion, 3D SDF surface regression, and 5D radiance field reconstruction.
  arXiv  Detail & Related papers  (2022-05-26T17:43:03Z)
• Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time [43.0484840009621]
  Implicit neural representations such as Neural Radiance Field (NeRF) have focused mainly on modeling static objects captured under multi-view settings. We present a novel Fourier PlenOctree (FPO) technique to tackle efficient neural modeling and real-time rendering of dynamic scenes captured under the free-view video (FVV) setting. We show that the proposed method is 3000 times faster than the original NeRF and over an order of magnitude acceleration over SOTA.
  arXiv  Detail & Related papers  (2022-02-17T11:57:01Z)
• Functional Regularization for Reinforcement Learning via Learned Fourier Features [98.90474131452588]
  We propose a simple architecture for deep reinforcement learning by embedding inputs into a learned Fourier basis. We show that it improves the sample efficiency of both state-based and image-based RL.
  arXiv  Detail & Related papers  (2021-12-06T18:59:52Z)
• NeRF in detail: Learning to sample for view synthesis [104.75126790300735]
  Neural radiance fields (NeRF) methods have demonstrated impressive novel view synthesis. In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a performance and not trained end-to-end for the task at hand. We introduce a differentiable module that learns to propose samples and their importance for the fine network, and consider and compare multiple alternatives for its neural architecture.
  arXiv  Detail & Related papers  (2021-06-09T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.