Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance

• URL: http://arxiv.org/abs/2605.06214v1
• Date: Thu, 07 May 2026 13:15:48 GMT
• Title: Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance
• Abstract summary: We present a differentiable framework to adaptively compute 4D illumination conditions with respect to an object.<n>We use a simple histogram-based pixel-level probability model for depth and reflectance.<n>As new structured illumination is cast, corresponding image measurements are used to update the uncertainty at each pixel.
• Score: 20.2007204470242
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a differentiable framework to adaptively compute 4D illumination conditions with respect to an object, for efficient, high-quality simultaneous acquisition of its shape and reflectance, with a unified spatial-angular structured light and a single camera. Using a simple histogram-based pixel-level probability model for depth and reflectance, we differentiably link the next illumination condition(s) with a loss that encourages the reduction in depth uncertainty. As new structured illumination is cast, corresponding image measurements are used to update the uncertainty at each pixel. Finally, a fine-tuning-based approach reconstructs the depth map and reflectance parameter maps, by minimizing the differences between all physical measurements and their simulated counterparts. The effectiveness of our framework is demonstrated on physical objects with wide variations in shape and appearance. Our depth results compare favorably with state-of-the-art techniques, while our reflectance results are comparable when validated against photographs.

Related papers

• Under One Sun: Multi-Object Generative Perception of Materials and Illumination [24.59498855820178]
  We introduce Multi-Object Generative Perception (MultiGP), a generative inverse method for rendering all radiometric constituents -- reflectance, texture, and illumination -- underlying object appearance from a single image.<n>Our key idea to solve this inherently ambiguous radiometric disentanglement is to leverage the fact that while their texture and reflectance may differ, objects in the same scene are all lit by the same illumination.<n> Experimental results demonstrate that MultiGP effectively leverages the complementary spatial and frequency characteristics of multiple object appearances to recover individual texture and reflectance as well as the common illumination.
  arXiv  Detail & Related papers  (2026-03-19T17:59:45Z)
• Neural Multi-View Self-Calibrated Photometric Stereo without Photometric Stereo Cues [8.270913200307197]
  We propose a neural inverse rendering approach that jointly reconstructs geometry, spatially varying reflectance, and lighting conditions from multi-view images.<n>We represent both geometry and reflectance as neural implicit fields and apply shadow-aware volume rendering.<n>The proposed method outperforms state-of-the-art normal-guided approaches in shape and lighting estimation accuracy.
  arXiv  Detail & Related papers  (2025-07-30T23:56:38Z)
• Acquisition of Spatially-Varying Reflectance and Surface Normals via Polarized Reflectance Fields [15.653977591138682]
  Accurately measuring the geometry and spatially-varying reflectance of real-world objects is a complex task.<n>We propose a novel approach using polarized reflectance field capture and a comprehensive statistical analysis algorithm.<n>We showcase the captured shapes and reflectance of diverse objects with a wide material range, spanning from highly diffuse to highly glossy.
  arXiv  Detail & Related papers  (2024-12-13T00:39:55Z)
• Adaptive Stereo Depth Estimation with Multi-Spectral Images Across All Lighting Conditions [58.88917836512819]
  We propose a novel framework incorporating stereo depth estimation to enforce accurate geometric constraints. To mitigate the effects of poor lighting on stereo matching, we introduce Degradation Masking. Our method achieves state-of-the-art (SOTA) performance on the Multi-Spectral Stereo (MS2) dataset.
  arXiv  Detail & Related papers  (2024-11-06T03:30:46Z)
• DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering [75.86523223933912]
  Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by the unknown light. We propose DANI-Net, an inverse rendering framework with differentiable shadow handling and anisotropic reflectance modeling. Our network benefits from cues of shadow and anisotropic reflectance through two differentiable paths.
  arXiv  Detail & Related papers  (2023-03-27T11:10:54Z)
• Multi-View Neural Surface Reconstruction with Structured Light [7.709526244898887]
  Three-dimensional (3D) object reconstruction based on differentiable rendering (DR) is an active research topic in computer vision. We introduce active sensing with structured light (SL) into multi-view 3D object reconstruction based on DR to learn the unknown geometry and appearance of arbitrary scenes and camera poses. Our method realizes high reconstruction accuracy in the textureless region and reduces efforts for camera pose calibration.
  arXiv  Detail & Related papers  (2022-11-22T03:10:46Z)
• Self-calibrating Photometric Stereo by Neural Inverse Rendering [88.67603644930466]
  This paper tackles the task of uncalibrated photometric stereo for 3D object reconstruction. We propose a new method that jointly optimize object shape, light directions, and light intensities. Our method demonstrates state-of-the-art accuracy in light estimation and shape recovery on real-world datasets.
  arXiv  Detail & Related papers  (2022-07-16T02:46:15Z)
• Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo [61.6260594326246]
  We introduce an efficient fully-convolutional architecture that can leverage both spatial and photometric context simultaneously. Using separable 4D convolutions and 2D heat-maps reduces the size and makes more efficient.
  arXiv  Detail & Related papers  (2021-03-22T18:06:58Z)
• Category Level Object Pose Estimation via Neural Analysis-by-Synthesis [64.14028598360741]
  In this paper we combine a gradient-based fitting procedure with a parametric neural image synthesis module. The image synthesis network is designed to efficiently span the pose configuration space. We experimentally show that the method can recover orientation of objects with high accuracy from 2D images alone.
  arXiv  Detail & Related papers  (2020-08-18T20:30:47Z)
• Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
  We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object. We first estimate per-view depth maps using a deep multi-view stereo network. These depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture.
  arXiv  Detail & Related papers  (2020-03-27T21:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.