Differentiable Display Photometric Stereo

• URL: http://arxiv.org/abs/2306.13325v4
• Date: Tue, 12 Mar 2024 12:21:18 GMT
• Title: Differentiable Display Photometric Stereo
• Authors: Seokjun Choi, Seungwoo Yoon, Giljoo Nam, Seungyong Lee, Seung-Hwan Baek
• Abstract summary: Photometric stereo leverages variations in illumination conditions to reconstruct surface normals. We present differentiable display photometric stereo (DDPS), addressing the design of display patterns. DDPS learns the display patterns that yield accurate normal reconstruction for a target system in an end-to-end manner.
• Score: 15.842538322034537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Photometric stereo leverages variations in illumination conditions to reconstruct surface normals. Display photometric stereo, which employs a conventional monitor as an illumination source, has the potential to overcome limitations often encountered in bulky and difficult-to-use conventional setups. In this paper, we present differentiable display photometric stereo (DDPS), addressing an often overlooked challenge in display photometric stereo: the design of display patterns. Departing from using heuristic display patterns, DDPS learns the display patterns that yield accurate normal reconstruction for a target system in an end-to-end manner. To this end, we propose a differentiable framework that couples basis-illumination image formation with analytic photometric-stereo reconstruction. The differentiable framework facilitates the effective learning of display patterns via auto-differentiation. Also, for training supervision, we propose to use 3D printing for creating a real-world training dataset, enabling accurate reconstruction on the target real-world setup. Finally, we exploit that conventional LCD monitors emit polarized light, which allows for the optical separation of diffuse and specular reflections when combined with a polarization camera, leading to accurate normal reconstruction. Extensive evaluation of DDPS shows improved normal-reconstruction accuracy compared to heuristic patterns and demonstrates compelling properties such as robustness to pattern initialization, calibration errors, and simplifications in image formation and reconstruction.

Related papers

• Differentiable Mobile Display Photometric Stereo [8.436027244967438]
  Display photometric stereo uses a display as a programmable light source to illuminate a scene with diverse illumination conditions. Recently, differentiable display photometric stereo demonstrated improved normal reconstruction accuracy by using learned display patterns. We propose a more practical physics-based photometric stereo, differentiable mobile display photometric stereo.
  arXiv  Detail & Related papers  (2025-02-07T16:24:56Z)
• PRM: Photometric Stereo based Large Reconstruction Model [18.95196627599738]
  PRM renders photometric stereo images by varying materials and lighting for the purposes. To offer enhanced flexibility of images rendering, we incorporate a real-time physically-based rendering (PBR) method and mesh specularization for online images.
  arXiv  Detail & Related papers  (2024-12-10T10:11:15Z)
• MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo [26.032964551717548]
  Photometric stereo typically demands intricate data acquisition setups involving multiple light sources to recover surface normals accurately. We propose MERLiN, an attention-based hourglass network that integrates single image-based inverse rendering and relighting within a single unified framework.
  arXiv  Detail & Related papers  (2024-09-01T09:32:03Z)
• Learning Photometric Feature Transform for Free-form Object Scan [44.21941847090453]
  We propose a novel framework to automatically learn to aggregate and transform photometric measurements from unstructured views. We build a system to reconstruct both the geometry and anisotropic reflectance of a variety of challenging objects from hand-held scans. Results are validated against reconstructions from a professional 3D scanner and photographs, and compare favorably with state-of-the-art techniques.
  arXiv  Detail & Related papers  (2023-08-07T11:34:27Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• How to Calibrate Your Event Camera [58.80418612800161]
  We propose a generic event camera calibration framework using image reconstruction. We show that neural-network-based image reconstruction is well suited for the task of intrinsic and extrinsic calibration of event cameras.
  arXiv  Detail & Related papers  (2021-05-26T07:06:58Z)
• SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses [82.56853587380168]
  We propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of the same scene from different lens should be the same. We leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters. Our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-30T08:23:25Z)
• Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo [52.68109922159688]
  We introduce a novel differentiable image formation model for active stereo, relying on both wave and geometric optics, and a novel trinocular reconstruction network. The jointly optimized pattern, which we dub "Polka Lines," together with the reconstruction network, achieve state-of-the-art active-stereo depth estimates across imaging conditions.
  arXiv  Detail & Related papers  (2020-11-26T04:02:43Z)
• Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
  We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object. We first estimate per-view depth maps using a deep multi-view stereo network. These depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture.
  arXiv  Detail & Related papers  (2020-03-27T21:28:54Z)
• Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials [65.95928593628128]
  We present a method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo technique. Our algorithm is suitable for perspective cameras and nearby point light sources.
  arXiv  Detail & Related papers  (2020-01-18T12:26:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.