Self-calibrating Photometric Stereo by Neural Inverse Rendering

• URL: http://arxiv.org/abs/2207.07815v1
• Date: Sat, 16 Jul 2022 02:46:15 GMT
• Title: Self-calibrating Photometric Stereo by Neural Inverse Rendering
• Authors: Junxuan Li and Hongdong Li
• Abstract summary: This paper tackles the task of uncalibrated photometric stereo for 3D object reconstruction. We propose a new method that jointly optimize object shape, light directions, and light intensities. Our method demonstrates state-of-the-art accuracy in light estimation and shape recovery on real-world datasets.
• Score: 88.67603644930466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper tackles the task of uncalibrated photometric stereo for 3D object reconstruction, where both the object shape, object reflectance, and lighting directions are unknown. This is an extremely difficult task, and the challenge is further compounded with the existence of the well-known generalized bas-relief (GBR) ambiguity in photometric stereo. Previous methods to resolve this ambiguity either rely on an overly simplified reflectance model, or assume special light distribution. We propose a new method that jointly optimizes object shape, light directions, and light intensities, all under general surfaces and lights assumptions. The specularities are used explicitly to solve uncalibrated photometric stereo via a neural inverse rendering process. We gradually fit specularities from shiny to rough using novel progressive specular bases. Our method leverages a physically based rendering equation by minimizing the reconstruction error on a per-object-basis. Our method demonstrates state-of-the-art accuracy in light estimation and shape recovery on real-world datasets.

Related papers

• MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo [26.032964551717548]
  Photometric stereo typically demands intricate data acquisition setups involving multiple light sources to recover surface normals accurately. We propose MERLiN, an attention-based hourglass network that integrates single image-based inverse rendering and relighting within a single unified framework.
  arXiv  Detail & Related papers  (2024-09-01T09:32:03Z)
• Deep Learning Methods for Calibrated Photometric Stereo and Beyond [86.57469194387264]
  Photometric stereo recovers the surface normals of an object from multiple images with varying shading cues. Deep learning methods have shown a powerful ability in the context of photometric stereo against non-Lambertian surfaces.
  arXiv  Detail & Related papers  (2022-12-16T11:27:44Z)
• A CNN Based Approach for the Point-Light Photometric Stereo Problem [26.958763133729846]
  We propose a CNN-based approach capable of handling realistic assumptions by leveraging recent improvements of deep neural networks for far-field Photometric Stereo. Our approach outperforms the state-of-the-art on the DiLiGenT real world dataset. In order to measure the performance of our approach for near-field point-light source PS data, we introduce LUCES the first real-world 'dataset for near-fieLd point light soUrCe photomEtric Stereo'
  arXiv  Detail & Related papers  (2022-10-10T12:57:12Z)
• Edge-preserving Near-light Photometric Stereo with Neural Surfaces [76.50065919656575]
  We introduce an analytically differentiable neural surface in near-light photometric stereo for avoiding differentiation errors at sharp depth edges. Experiments on both synthetic and real-world scenes demonstrate the effectiveness of our method for detailed shape recovery with edge preservation.
  arXiv  Detail & Related papers  (2022-07-11T04:51:43Z)
• Universal Photometric Stereo Network using Global Lighting Contexts [4.822598110892846]
  This paper tackles a new photometric stereo task, named universal photometric stereo. It is supposed to work for objects with diverse shapes and materials under arbitrary lighting variations without assuming any specific models.
  arXiv  Detail & Related papers  (2022-06-06T09:32:06Z)
• Multi-view 3D Reconstruction of a Texture-less Smooth Surface of Unknown Generic Reflectance [86.05191217004415]
  Multi-view reconstruction of texture-less objects with unknown surface reflectance is a challenging task. This paper proposes a simple and robust solution to this problem based on a co-light scanner.
  arXiv  Detail & Related papers  (2021-05-25T01:28:54Z)
• Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo [61.6260594326246]
  We introduce an efficient fully-convolutional architecture that can leverage both spatial and photometric context simultaneously. Using separable 4D convolutions and 2D heat-maps reduces the size and makes more efficient.
  arXiv  Detail & Related papers  (2021-03-22T18:06:58Z)
• Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces [103.08512487830669]
  This paper presents an uncalibrated deep neural network framework for the photometric stereo problem. Existing neural network-based methods either require exact light directions or ground-truth surface normals of the object or both. We propose an uncalibrated neural inverse rendering approach to this problem.
  arXiv  Detail & Related papers  (2020-12-12T10:33:08Z)
• A CNN Based Approach for the Near-Field Photometric Stereo Problem [26.958763133729846]
  We propose the first CNN based approach capable of handling realistic assumptions in Photometric Stereo. We leverage recent improvements of deep neural networks for far-field Photometric Stereo and adapt them to near field setup. Our method outperforms competing state-of-the-art near-field Photometric Stereo approaches on both synthetic and real experiments.
  arXiv  Detail & Related papers  (2020-09-12T13:28:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.