Polarimetric Multi-View Inverse Rendering

• URL: http://arxiv.org/abs/2212.12721v1
• Date: Sat, 24 Dec 2022 12:12:12 GMT
• Title: Polarimetric Multi-View Inverse Rendering
• Authors: Jinyu Zhao, Yusuke Monno, Masatoshi Okutomi
• Abstract summary: A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) and the degree of polarization (DoP) of reflected light are related to an object's surface normal. We propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color-polarization images.
• Score: 13.391866136230165
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) and the degree of polarization (DoP) of reflected light are related to an object's surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color-polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view stereo pipeline. We then refine the initial model by optimizing photometric rendering errors and polarimetric errors using multi-view RGB, AoP, and DoP images, where we propose a novel polarimetric cost function that enables an effective constraint on the estimated surface normal of each vertex, while considering four possible ambiguous azimuth angles revealed from the AoP measurement. The weight for the polarimetric cost is effectively determined based on the DoP measurement, which is regarded as the reliability of polarimetric information. Experimental results using both synthetic and real data demonstrate that our Polarimetric MVIR can reconstruct a detailed 3D shape without assuming a specific surface material and lighting condition.

Related papers

• NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images [62.752710734332894]
  NeRSP is a Neural 3D reconstruction technique for Reflective surfaces with Sparse Polarized images. We derive photometric and geometric cues from the polarimetric image formation model and multiview azimuth consistency. We achieve the state-of-the-art surface reconstruction results with only 6 views as input.
  arXiv  Detail & Related papers  (2024-06-11T09:53:18Z)
• S2P3: Self-Supervised Polarimetric Pose Prediction [55.43547228561919]
  This paper proposes the first self-supervised 6D object pose prediction from multimodal RGB+polarimetric images. The novel training paradigm comprises 1) a physical model to extract geometric information of polarized light, 2) a teacher-student knowledge distillation scheme and 3) a self-supervised loss formulation through differentiable constraints.
  arXiv  Detail & Related papers  (2023-12-02T10:46:40Z)
• Sparse Ellipsometry: Portable Acquisition of Polarimetric SVBRDF and Shape with Unstructured Flash Photography [32.68190169944569]
  We present a portable polarimetric acquisition method that captures both polarimetric SVBRDF and 3D shape simultaneously. Instead of days, the total acquisition time varies between twenty and thirty minutes per object. Our results show a strong agreement with a recent ground-truth dataset of captured polarimetric BRDFs of real-world objects.
  arXiv  Detail & Related papers  (2022-07-09T09:42:59Z)
• PolarFormer: Multi-camera 3D Object Detection with Polar Transformers [93.49713023975727]
  3D object detection in autonomous driving aims to reason "what" and "where" the objects of interest present in a 3D world. Existing methods often adopt the canonical Cartesian coordinate system with perpendicular axis. We propose a new Polar Transformer (PolarFormer) for more accurate 3D object detection in the bird's-eye-view (BEV) taking as input only multi-camera 2D images.
  arXiv  Detail & Related papers  (2022-06-30T16:32:48Z)
• Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo [103.08512487830669]
  We present a modern solution to the multi-view photometric stereo problem (MVPS) We procure the surface orientation using a photometric stereo (PS) image formation model and blend it with a multi-view neural radiance field representation to recover the object's surface geometry. Our method performs neural rendering of multi-view images while utilizing surface normals estimated by a deep photometric stereo network.
  arXiv  Detail & Related papers  (2021-10-11T20:20:03Z)
• Human Pose and Shape Estimation from Single Polarization Images [45.24275141578927]
  We attempt to estimate human pose and shape from single polarization images by leveraging the polarization-induced geometric cues. A dedicated dataset (PHSPD) is constructed, consisting of over 500K frames with accurate pose and shape annotations. It suggests polarization camera as a promising alternative to the more conventional RGB camera for human pose and shape estimation.
  arXiv  Detail & Related papers  (2021-08-15T22:56:18Z)
• Deep Polarization Imaging for 3D shape and SVBRDF Acquisition [7.86578678811226]
  We present a novel method for efficient acquisition of shape and spatially varying reflectance of 3D objects using polarization cues. Unlike previous works that have exploited polarization to estimate material or object appearance under certain constraints, we lift such restrictions by coupling polarization imaging with deep learning. We demonstrate our approach to achieve superior results compared to recent works employing deep learning in conjunction with flash illumination.
  arXiv  Detail & Related papers  (2021-05-06T17:58:43Z)
• Polarimetric Multi-View Inverse Rendering [13.391866136230165]
  A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) of reflected light is related to an object's surface normal. We propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that exploits geometric, photometric, and polarimetric cues extracted from input multi-view color polarization images.
  arXiv  Detail & Related papers  (2020-07-17T09:00:20Z)
• Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials [65.95928593628128]
  We present a method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo technique. Our algorithm is suitable for perspective cameras and nearby point light sources.
  arXiv  Detail & Related papers  (2020-01-18T12:26:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.