SNeS: Learning Probably Symmetric Neural Surfaces from Incomplete Data
- URL: http://arxiv.org/abs/2206.06340v1
- Date: Mon, 13 Jun 2022 17:37:50 GMT
- Title: SNeS: Learning Probably Symmetric Neural Surfaces from Incomplete Data
- Authors: Eldar Insafutdinov, Dylan Campbell, Jo\~ao F. Henriques, Andrea
Vedaldi
- Abstract summary: We build on the strengths of recent advances in neural reconstruction and rendering such as Neural Radiance Fields (NeRF)
We apply a soft symmetry constraint to the 3D geometry and material properties, having factored appearance into lighting, albedo colour and reflectivity.
We show that it can reconstruct unobserved regions with high fidelity and render high-quality novel view images.
- Score: 77.53134858717728
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We present a method for the accurate 3D reconstruction of partly-symmetric
objects. We build on the strengths of recent advances in neural reconstruction
and rendering such as Neural Radiance Fields (NeRF). A major shortcoming of
such approaches is that they fail to reconstruct any part of the object which
is not clearly visible in the training image, which is often the case for
in-the-wild images and videos. When evidence is lacking, structural priors such
as symmetry can be used to complete the missing information. However,
exploiting such priors in neural rendering is highly non-trivial: while
geometry and non-reflective materials may be symmetric, shadows and reflections
from the ambient scene are not symmetric in general. To address this, we apply
a soft symmetry constraint to the 3D geometry and material properties, having
factored appearance into lighting, albedo colour and reflectivity. We evaluate
our method on the recently introduced CO3D dataset, focusing on the car
category due to the challenge of reconstructing highly-reflective materials. We
show that it can reconstruct unobserved regions with high fidelity and render
high-quality novel view images.
Related papers
- UniSDF: Unifying Neural Representations for High-Fidelity 3D
Reconstruction of Complex Scenes with Reflections [92.38975002642455]
We propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections.
Our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces.
arXiv Detail & Related papers (2023-12-20T18:59:42Z) - A Perceptual Shape Loss for Monocular 3D Face Reconstruction [13.527078921914985]
We propose a new loss function for monocular face capture inspired by how humans would perceive the quality of a 3D face reconstruction.
Our loss is implemented as a discriminator-style neural network that takes an input face image and a shaded render of the geometry estimate.
We show how our new perceptual shape loss can be combined with traditional energy terms for monocular 3D face optimization and deep neural network regression.
arXiv Detail & Related papers (2023-10-30T14:39:11Z) - Seeing Through the Glass: Neural 3D Reconstruction of Object Inside a
Transparent Container [61.50401406132946]
Transparent enclosures pose challenges of multiple light reflections and refractions at the interface between different propagation media.
We use an existing neural reconstruction method (NeuS) that implicitly represents the geometry and appearance of the inner subspace.
In order to account for complex light interactions, we develop a hybrid rendering strategy that combines volume rendering with ray tracing.
arXiv Detail & Related papers (2023-03-24T04:58:27Z) - SIRA: Relightable Avatars from a Single Image [19.69326772087838]
We introduce SIRA, a method which reconstructs human head avatars with high fidelity geometry and factorized lights and surface materials.
Our key ingredients are two data-driven statistical models based on neural fields that resolve the ambiguities of single-view 3D surface reconstruction and appearance factorization.
arXiv Detail & Related papers (2022-09-07T09:47:46Z) - Neural Reflectance for Shape Recovery with Shadow Handling [88.67603644930466]
This paper aims at recovering the shape of a scene with unknown, non-Lambertian, and possibly spatially-varying surface materials.
We propose a coordinate-based deep reflectance (multilayer perceptron) to parameterize both the unknown 3D shape and the unknown at every surface point.
This network is able to leverage the observed photometric variance and shadows on the surface, and recover both surface shape and general non-Lambertian reflectance.
arXiv Detail & Related papers (2022-03-24T07:57:20Z) - NeRFactor: Neural Factorization of Shape and Reflectance Under an
Unknown Illumination [60.89737319987051]
We address the problem of recovering shape and spatially-varying reflectance of an object from posed multi-view images of the object illuminated by one unknown lighting condition.
This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties.
arXiv Detail & Related papers (2021-06-03T16:18:01Z) - Multi-view 3D Reconstruction of a Texture-less Smooth Surface of Unknown
Generic Reflectance [86.05191217004415]
Multi-view reconstruction of texture-less objects with unknown surface reflectance is a challenging task.
This paper proposes a simple and robust solution to this problem based on a co-light scanner.
arXiv Detail & Related papers (2021-05-25T01:28:54Z) - NeRD: Neural 3D Reflection Symmetry Detector [27.626579746101292]
We present NeRD, a Neural 3D Reflection Symmetry Detector.
We first enumerate the symmetry planes with a coarse-to-fine strategy and then find the best ones by building 3D cost volumes.
Our experiments show that the symmetry planes detected with our method are significantly more accurate than the planes from direct CNN regression.
arXiv Detail & Related papers (2021-04-19T17:25:51Z) - Shape From Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF
Material from Images via Differentiable Path Tracing [16.975014467319443]
Differentiable path tracing is an appealing framework as it can reproduce complex appearance effects.
We show how to use differentiable ray tracing to refine an initial coarse mesh and per-mesh-facet material representation.
We also show how to refine initial reconstructions of real-world objects in unconstrained environments.
arXiv Detail & Related papers (2020-12-06T18:55:35Z) - Learning to Detect 3D Reflection Symmetry for Single-View Reconstruction [32.14605731030579]
3D reconstruction from a single RGB image is a challenging problem in computer vision.
Previous methods are usually solely data-driven, which lead to inaccurate 3D shape recovery and limited generalization capability.
We present a geometry-based end-to-end deep learning framework that first detects the mirror plane of reflection symmetry that commonly exists in man-made objects and then predicts depth maps by finding the intra-image pixel-wise correspondence of the symmetry.
arXiv Detail & Related papers (2020-06-17T17:58:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.