Learning Implicit Surface Light Fields
- URL: http://arxiv.org/abs/2003.12406v1
- Date: Fri, 27 Mar 2020 13:17:45 GMT
- Title: Learning Implicit Surface Light Fields
- Authors: Michael Oechsle, Michael Niemeyer, Lars Mescheder, Thilo Strauss,
Andreas Geiger
- Abstract summary: Implicit representations of 3D objects have recently achieved impressive results on learning-based 3D reconstruction tasks.
We propose a novel implicit representation for capturing the visual appearance of an object in terms of its surface light field.
Our model is able to infer rich visual appearance including shadows and specular reflections.
- Score: 34.89812112073539
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Implicit representations of 3D objects have recently achieved impressive
results on learning-based 3D reconstruction tasks. While existing works use
simple texture models to represent object appearance, photo-realistic image
synthesis requires reasoning about the complex interplay of light, geometry and
surface properties. In this work, we propose a novel implicit representation
for capturing the visual appearance of an object in terms of its surface light
field. In contrast to existing representations, our implicit model represents
surface light fields in a continuous fashion and independent of the geometry.
Moreover, we condition the surface light field with respect to the location and
color of a small light source. Compared to traditional surface light field
models, this allows us to manipulate the light source and relight the object
using environment maps. We further demonstrate the capabilities of our model to
predict the visual appearance of an unseen object from a single real RGB image
and corresponding 3D shape information. As evidenced by our experiments, our
model is able to infer rich visual appearance including shadows and specular
reflections. Finally, we show that the proposed representation can be embedded
into a variational auto-encoder for generating novel appearances that conform
to the specified illumination conditions.
Related papers
- BiGS: Bidirectional Gaussian Primitives for Relightable 3D Gaussian Splatting [10.918133974256913]
We present Bidirectional Gaussian Primitives, an image-based novel view synthesis technique.
Our approach integrates light intrinsic decomposition into the Gaussian splatting framework, enabling real-time relighting of 3D objects.
arXiv Detail & Related papers (2024-08-23T21:04:40Z) - Ghost on the Shell: An Expressive Representation of General 3D Shapes [97.76840585617907]
Meshes are appealing since they enable fast physics-based rendering with realistic material and lighting.
Recent work on reconstructing and statistically modeling 3D shapes has critiqued meshes as being topologically inflexible.
We parameterize open surfaces by defining a manifold signed distance field on watertight surfaces.
G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks.
arXiv Detail & Related papers (2023-10-23T17:59:52Z) - NEMTO: Neural Environment Matting for Novel View and Relighting Synthesis of Transparent Objects [28.62468618676557]
We propose NEMTO, the first end-to-end neural rendering pipeline to model 3D transparent objects.
With 2D images of the transparent object as input, our method is capable of high-quality novel view and relighting synthesis.
arXiv Detail & Related papers (2023-03-21T15:50:08Z) - Physics-based Indirect Illumination for Inverse Rendering [70.27534648770057]
We present a physics-based inverse rendering method that learns the illumination, geometry, and materials of a scene from posed multi-view RGB images.
As a side product, our physics-based inverse rendering model also facilitates flexible and realistic material editing as well as relighting.
arXiv Detail & Related papers (2022-12-09T07:33:49Z) - Generative Deformable Radiance Fields for Disentangled Image Synthesis
of Topology-Varying Objects [52.46838926521572]
3D-aware generative models have demonstrated their superb performance to generate 3D neural radiance fields (NeRF) from a collection of monocular 2D images.
We propose a generative model for synthesizing radiance fields of topology-varying objects with disentangled shape and appearance variations.
arXiv Detail & Related papers (2022-09-09T08:44:06Z) - Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image.
Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene.
We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
arXiv Detail & Related papers (2021-09-13T15:29:03Z) - Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images [64.53227129573293]
We investigate the problem of learning to generate 3D parametric surface representations for novel object instances, as seen from one or more views.
We design neural networks capable of generating high-quality parametric 3D surfaces which are consistent between views.
Our method is supervised and trained on a public dataset of shapes from common object categories.
arXiv Detail & Related papers (2020-08-18T06:33:40Z) - Neural Reflectance Fields for Appearance Acquisition [61.542001266380375]
We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene.
We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
arXiv Detail & Related papers (2020-08-09T22:04:36Z) - IllumiNet: Transferring Illumination from Planar Surfaces to Virtual
Objects in Augmented Reality [38.83696624634213]
This paper presents an illumination estimation method for virtual objects in real environment by learning.
Given a single RGB image, our method directly infers the relit virtual object by transferring the illumination features extracted from planar surfaces in the scene to the desired geometries.
arXiv Detail & Related papers (2020-07-12T13:11:14Z) - Multiview Neural Surface Reconstruction by Disentangling Geometry and
Appearance [46.488713939892136]
We introduce a neural network that simultaneously learns the unknown geometry, camera parameters, and a neural architecture that approximates the light reflected from the surface towards the camera.
We trained our network on real world 2D images of objects with different material properties, lighting conditions, and noisy camera materials from the DTU MVS dataset.
arXiv Detail & Related papers (2020-03-22T10:20:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.