Differentiable Rendering with Reparameterized Volume Sampling
- URL: http://arxiv.org/abs/2302.10970v3
- Date: Sat, 2 Mar 2024 00:31:18 GMT
- Title: Differentiable Rendering with Reparameterized Volume Sampling
- Authors: Nikita Morozov, Denis Rakitin, Oleg Desheulin, Dmitry Vetrov, Kirill
Struminsky
- Abstract summary: In view synthesis, a neural radiance field approximates underlying density and radiance fields based on a sparse set of scene pictures.
This rendering algorithm is fully differentiable and facilitates gradient-based optimization of the fields.
We propose a simple end-to-end differentiable sampling algorithm based on inverse transform sampling.
- Score: 2.717399369766309
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In view synthesis, a neural radiance field approximates underlying density
and radiance fields based on a sparse set of scene pictures. To generate a
pixel of a novel view, it marches a ray through the pixel and computes a
weighted sum of radiance emitted from a dense set of ray points. This rendering
algorithm is fully differentiable and facilitates gradient-based optimization
of the fields. However, in practice, only a tiny opaque portion of the ray
contributes most of the radiance to the sum. We propose a simple end-to-end
differentiable sampling algorithm based on inverse transform sampling. It
generates samples according to the probability distribution induced by the
density field and picks non-transparent points on the ray. We utilize the
algorithm in two ways. First, we propose a novel rendering approach based on
Monte Carlo estimates. This approach allows for evaluating and optimizing a
neural radiance field with just a few radiance field calls per ray. Second, we
use the sampling algorithm to modify the hierarchical scheme proposed in the
original NeRF work. We show that our modification improves reconstruction
quality of hierarchical models, at the same time simplifying the training
procedure by removing the need for auxiliary proposal network losses.
Related papers
- Flash Cache: Reducing Bias in Radiance Cache Based Inverse Rendering [62.92985004295714]
We present a method that avoids approximations that introduce bias into the renderings and, more importantly, the gradients used for optimization.
We show that by removing these biases our approach improves the generality of radiance cache based inverse rendering, as well as increasing quality in the presence of challenging light transport effects such as specular reflections.
arXiv Detail & Related papers (2024-09-09T17:59:57Z) - RayGauss: Volumetric Gaussian-Based Ray Casting for Photorealistic Novel View Synthesis [3.4341938551046227]
Differentiable rendering methods made significant progress in novel view synthesis.
We provide a consistent formulation of the emitted radiance c and density sigma for differentiable ray casting of irregularly distributed Gaussians.
We achieve superior quality rendering compared to the state-of-the-art while maintaining reasonable training times and achieving inference speeds of 25 FPS on the Blender dataset.
arXiv Detail & Related papers (2024-08-06T10:59:58Z) - CVT-xRF: Contrastive In-Voxel Transformer for 3D Consistent Radiance Fields from Sparse Inputs [65.80187860906115]
We propose a novel approach to improve NeRF's performance with sparse inputs.
We first adopt a voxel-based ray sampling strategy to ensure that the sampled rays intersect with a certain voxel in 3D space.
We then randomly sample additional points within the voxel and apply a Transformer to infer the properties of other points on each ray, which are then incorporated into the volume rendering.
arXiv Detail & Related papers (2024-03-25T15:56:17Z) - Neural Fields with Thermal Activations for Arbitrary-Scale Super-Resolution [56.089473862929886]
We present a novel way to design neural fields such that points can be queried with an adaptive Gaussian PSF.
With its theoretically guaranteed anti-aliasing, our method sets a new state of the art for arbitrary-scale single image super-resolution.
arXiv Detail & Related papers (2023-11-29T14:01:28Z) - Adaptive Shells for Efficient Neural Radiance Field Rendering [92.18962730460842]
We propose a neural radiance formulation that smoothly transitions between- and surface-based rendering.
Our approach enables efficient rendering at very high fidelity.
We also demonstrate that the extracted envelope enables downstream applications such as animation and simulation.
arXiv Detail & Related papers (2023-11-16T18:58:55Z) - Efficient Ray Sampling for Radiance Fields Reconstruction [4.004168836949491]
ray sampling strategy profoundly impacts network convergence.
We propose a novel ray sampling approach for neural radiance fields.
Our method significantly outperforms state-of-the-art techniques on public benchmark datasets.
arXiv Detail & Related papers (2023-08-29T18:11:32Z) - Estimating Neural Reflectance Field from Radiance Field using Tree
Structures [29.431165709718794]
We present a new method for estimating the Neural Reflectance Field (NReF) of an object from a set of posed multi-view images under unknown lighting.
NReF represents 3D geometry and appearance of objects in a disentangled manner, and are hard to be estimated from images only.
Our method solves this problem by exploiting the Neural Radiance Field (NeRF) as a proxy representation, from which we perform further decomposition.
arXiv Detail & Related papers (2022-10-09T10:21:31Z) - InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering [55.70938412352287]
We present an information-theoretic regularization technique for few-shot novel view synthesis based on neural implicit representation.
The proposed approach minimizes potential reconstruction inconsistency that happens due to insufficient viewpoints.
We achieve consistently improved performance compared to existing neural view synthesis methods by large margins on multiple standard benchmarks.
arXiv Detail & Related papers (2021-12-31T11:56:01Z) - Efficient and Differentiable Shadow Computation for Inverse Problems [64.70468076488419]
Differentiable geometric computation has received increasing interest for image-based inverse problems.
We propose an efficient yet efficient approach for differentiable visibility and soft shadow computation.
As our formulation is differentiable, it can be used to solve inverse problems such as texture, illumination, rigid pose, and deformation recovery from images.
arXiv Detail & Related papers (2021-04-01T09:29:05Z) - Contour Integration using Graph-Cut and Non-Classical Receptive Field [4.935491924643742]
We propose a novel method to detect image contours from the extracted edge segments of other algorithms.
The proposed energy functions are inspired by the surround modulation in the primary visual cortex that help suppressing texture noise.
arXiv Detail & Related papers (2020-10-27T19:07:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.