Neural Point Catacaustics for Novel-View Synthesis of Reflections
- URL: http://arxiv.org/abs/2301.01087v1
- Date: Tue, 3 Jan 2023 13:28:10 GMT
- Title: Neural Point Catacaustics for Novel-View Synthesis of Reflections
- Authors: Georgios Kopanas, Thomas Leimk\"uhler, Gilles Rainer, Cl\'ement
Jambon, George Drettakis
- Abstract summary: We introduce a new point-based representation to compute Neural Point Catacaustics allowing novel-view synthesis of scenes with curved reflectors.
We provide the source code and other supplemental material on https://repo-sam.inria.fr/ fungraph/neural_catacaustics/.
- Score: 3.5348690973777
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: View-dependent effects such as reflections pose a substantial challenge for
image-based and neural rendering algorithms. Above all, curved reflectors are
particularly hard, as they lead to highly non-linear reflection flows as the
camera moves. We introduce a new point-based representation to compute Neural
Point Catacaustics allowing novel-view synthesis of scenes with curved
reflectors, from a set of casually-captured input photos. At the core of our
method is a neural warp field that models catacaustic trajectories of
reflections, so complex specular effects can be rendered using efficient point
splatting in conjunction with a neural renderer. One of our key contributions
is the explicit representation of reflections with a reflection point cloud
which is displaced by the neural warp field, and a primary point cloud which is
optimized to represent the rest of the scene. After a short manual annotation
step, our approach allows interactive high-quality renderings of novel views
with accurate reflection flow. Additionally, the explicit representation of
reflection flow supports several forms of scene manipulation in captured
scenes, such as reflection editing, cloning of specular objects, reflection
tracking across views, and comfortable stereo viewing. We provide the source
code and other supplemental material on https://repo-sam.inria.fr/
fungraph/neural_catacaustics/
Related papers
- Planar Reflection-Aware Neural Radiance Fields [32.709468082010126]
We introduce a reflection-aware NeRF that jointly models planar reflectors, such as windows, and explicitly casts reflected rays to capture the source of the high-frequency reflections.
Rendering along the primary ray results in a clean, reflection-free view, while explicitly rendering along the reflected ray allows us to reconstruct highly detailed reflections.
arXiv Detail & Related papers (2024-11-07T18:55:08Z) - NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections [57.63028964831785]
Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content.
We address these issues with an approach based on ray tracing.
Instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts rays from these points and traces them through the NeRF representation to render feature vectors.
arXiv Detail & Related papers (2024-05-23T17:59:57Z) - Learning Robust Multi-Scale Representation for Neural Radiance Fields
from Unposed Images [65.41966114373373]
We present an improved solution to the neural image-based rendering problem in computer vision.
The proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time.
arXiv Detail & Related papers (2023-11-08T08:18:23Z) - WildLight: In-the-wild Inverse Rendering with a Flashlight [77.31815397135381]
We propose a practical photometric solution for in-the-wild inverse rendering under unknown ambient lighting.
Our system recovers scene geometry and reflectance using only multi-view images captured by a smartphone.
We demonstrate by extensive experiments that our method is easy to implement, casual to set up, and consistently outperforms existing in-the-wild inverse rendering techniques.
arXiv Detail & Related papers (2023-03-24T17:59:56Z) - NeRFReN: Neural Radiance Fields with Reflections [16.28256369376256]
We introduce NeRFReN, which is built upon NeRF to model scenes with reflections.
We propose to split a scene into transmitted and reflected components, and model the two components with separate neural radiance fields.
Experiments on various self-captured scenes show that our method achieves high-quality novel view synthesis and physically sound depth estimation results.
arXiv Detail & Related papers (2021-11-30T09:36:00Z) - Free-viewpoint Indoor Neural Relighting from Multi-view Stereo [5.306819482496464]
We introduce a neural relighting algorithm for captured indoors scenes, that allows interactive free-viewpoint navigation.
Our method allows illumination to be changed synthetically, while coherently rendering cast shadows and complex glossy materials.
arXiv Detail & Related papers (2021-06-24T20:09:40Z) - Neural Reflectance Fields for Appearance Acquisition [61.542001266380375]
We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene.
We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
arXiv Detail & Related papers (2020-08-09T22:04:36Z) - Deep Reflectance Volumes: Relightable Reconstructions from Multi-View
Photometric Images [59.53382863519189]
We present a deep learning approach to reconstruct scene appearance from unstructured images captured under collocated point lighting.
At the heart of Deep Reflectance Volumes is a novel volumetric scene representation consisting of opacity, surface normal and reflectance voxel grids.
We show that our learned reflectance volumes are editable, allowing for modifying the materials of the captured scenes.
arXiv Detail & Related papers (2020-07-20T05:38:11Z) - Single image reflection removal via learning with multi-image
constraints [50.54095311597466]
We propose a novel learning-based solution that combines the advantages of the aforementioned approaches and overcomes their drawbacks.
Our algorithm works by learning a deep neural network to optimize the target with joint constraints enhanced among multiple input images.
Our algorithm runs in real-time and state-of-the-art reflection removal performance on real images.
arXiv Detail & Related papers (2019-12-08T06:10:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.