ORCa: Glossy Objects as Radiance Field Cameras

• URL: http://arxiv.org/abs/2212.04531v2
• Date: Mon, 12 Dec 2022 14:51:24 GMT
• Title: ORCa: Glossy Objects as Radiance Field Cameras
• Authors: Kushagra Tiwary, Akshat Dave, Nikhil Behari, Tzofi Klinghoffer, Ashok Veeraraghavan, Ramesh Raskar
• Abstract summary: We convert glossy objects with unknown geometry into radiance-field cameras to image the world from the object's perspective. We show that recovering the environment radiance fields enables depth and radiance estimation from the object to its surroundings. Our method is trained end-to-end on multi-view images of the object and jointly estimates object geometry, diffuse radiance, and the 5D environment radiance field.
• Score: 23.75324754684283
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reflections on glossy objects contain valuable and hidden information about the surrounding environment. By converting these objects into cameras, we can unlock exciting applications, including imaging beyond the camera's field-of-view and from seemingly impossible vantage points, e.g. from reflections on the human eye. However, this task is challenging because reflections depend jointly on object geometry, material properties, the 3D environment, and the observer viewing direction. Our approach converts glossy objects with unknown geometry into radiance-field cameras to image the world from the object's perspective. Our key insight is to convert the object surface into a virtual sensor that captures cast reflections as a 2D projection of the 5D environment radiance field visible to the object. We show that recovering the environment radiance fields enables depth and radiance estimation from the object to its surroundings in addition to beyond field-of-view novel-view synthesis, i.e. rendering of novel views that are only directly-visible to the glossy object present in the scene, but not the observer. Moreover, using the radiance field we can image around occluders caused by close-by objects in the scene. Our method is trained end-to-end on multi-view images of the object and jointly estimates object geometry, diffuse radiance, and the 5D environment radiance field.

Related papers

• Unsupervised Discovery of Object-Centric Neural Fields [21.223170092979498]
  We study inferring 3D object-centric scene representations from a single image. We propose Unsupervised discovery of Object-Centric neural Fields (uOCF)
  arXiv  Detail & Related papers  (2024-02-12T02:16:59Z)
• Virtual Mirrors: Non-Line-of-Sight Imaging Beyond the Third Bounce [11.767522056116842]
  Non-line-of-sight (NLOS) imaging methods are capable of reconstructing complex scenes that are not visible to an observer using indirect illumination. We make the key observation that planar diffuse surfaces behave specularly at wavelengths used in the computational wave-based NLOS imaging domain. We leverage this observation to expand the capabilities of NLOS imaging using illumination beyond the third bounce.
  arXiv  Detail & Related papers  (2023-07-26T17:59:20Z)
• NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images [44.1333444097976]
  We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment.
  arXiv  Detail & Related papers  (2023-05-27T07:40:07Z)
• Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes [62.769186261245416]
  We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows.
  arXiv  Detail & Related papers  (2023-04-06T17:51:54Z)
• Shadows Shed Light on 3D Objects [23.14510850163136]
  We create a differentiable image formation model that allows us to infer the 3D shape of an object, its pose, and the position of a light source. Our approach is robust to real-world images where ground-truth shadow mask is unknown.
  arXiv  Detail & Related papers  (2022-06-17T19:58:11Z)
• Unsupervised Discovery and Composition of Object Light Fields [57.198174741004095]
  We propose to represent objects in an object-centric, compositional scene representation as light fields. We propose a novel light field compositor module that enables reconstructing the global light field from a set of object-centric light fields.
  arXiv  Detail & Related papers  (2022-05-08T17:50:35Z)
• NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination [60.89737319987051]
  We address the problem of recovering shape and spatially-varying reflectance of an object from posed multi-view images of the object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties.
  arXiv  Detail & Related papers  (2021-06-03T16:18:01Z)
• Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenes [85.56602190773684]
  We build on the idea of view synthesis, which uses classical camera geometry to re-render a source image from a different point-of-view. By minimizing the error between the synthetic image and the corresponding real image in a video, the deep network that predicts pose and depth can be trained completely unsupervised.
  arXiv  Detail & Related papers  (2021-05-05T17:08:10Z)
• Refractive Light-Field Features for Curved Transparent Objects in Structure from Motion [10.380414189465345]
  We propose a novel image feature for light fields that detects and describes the patterns of light refracted through curved transparent objects. We demonstrate improved structure-from-motion performance in challenging scenes containing refractive objects. Our method is a critical step towards allowing robots to operate around refractive objects.
  arXiv  Detail & Related papers  (2021-03-29T05:55:32Z)
• Neural Reflectance Fields for Appearance Acquisition [61.542001266380375]
  We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene. We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
  arXiv  Detail & Related papers  (2020-08-09T22:04:36Z)
• Single View Metrology in the Wild [94.7005246862618]
  We present a novel approach to single view metrology that can recover the absolute scale of a scene represented by 3D heights of objects or camera height above the ground. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights. We demonstrate state-of-the-art qualitative and quantitative results on several datasets as well as applications including virtual object insertion.
  arXiv  Detail & Related papers  (2020-07-18T22:31:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.