Shape from Polarization for Complex Scenes in the Wild

• URL: http://arxiv.org/abs/2112.11377v1
• Date: Tue, 21 Dec 2021 17:30:23 GMT
• Title: Shape from Polarization for Complex Scenes in the Wild
• Authors: Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, Qifeng Chen
• Abstract summary: We present a new data-driven approach with physics-based priors to scene-level normal estimation from a single polarization image. We contribute the first real-world scene-level SfP dataset with paired input polarization images and ground-truth normal maps. Our trained model can be generalized to far-field outdoor scenes as the relationship between polarized light and surface normals is not affected by distance.
• Score: 93.65746187211958
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a new data-driven approach with physics-based priors to scene-level normal estimation from a single polarization image. Existing shape from polarization (SfP) works mainly focus on estimating the normal of a single object rather than complex scenes in the wild. A key barrier to high-quality scene-level SfP is the lack of real-world SfP data in complex scenes. Hence, we contribute the first real-world scene-level SfP dataset with paired input polarization images and ground-truth normal maps. Then we propose a learning-based framework with a multi-head self-attention module and viewing encoding, which is designed to handle increasing polarization ambiguities caused by complex materials and non-orthographic projection in scene-level SfP. Our trained model can be generalized to far-field outdoor scenes as the relationship between polarized light and surface normals is not affected by distance. Experimental results demonstrate that our approach significantly outperforms existing SfP models on two datasets. Our dataset and source code will be publicly available at \url{https://github.com/ChenyangLEI/sfp-wild}.

Related papers

• SS-SfP:Neural Inverse Rendering for Self Supervised Shape from (Mixed) Polarization [21.377923666134116]
  Shape from Polarization (SfP) is the problem popularly known as Shape from Polarization (SfP) We present a novel inverse rendering-based framework to estimate the 3D shape (per-pixel surface normals and depth) of objects and scenes from single-view polarization images.
  arXiv  Detail & Related papers  (2024-07-12T14:29:00Z)
• Video Frame Interpolation for Polarization via Swin-Transformer [9.10220649654041]
  Video Frame Interpolation (VFI) has been extensively explored and demonstrated, yet its application to polarization remains largely unexplored. This study proposes a multi-stage and multi-scale network called Swin-VFI based on the Swin-Transformer. Experimental results demonstrate our approach's superior reconstruction accuracy across all tasks.
  arXiv  Detail & Related papers  (2024-06-17T09:48:54Z)
• Towards Image Ambient Lighting Normalization [47.42834070783831]
  Ambient Lighting Normalization (ALN) enables the study of interactions between shadows, unifying image restoration and shadow removal in a broader context. For benchmarking, we select various mainstream methods and rigorously evaluate them on Ambient6K. Experiments show that IFBlend achieves SOTA scores on Ambient6K and exhibits competitive performance on conventional shadow removal benchmarks.
  arXiv  Detail & Related papers  (2024-03-27T16:20:55Z)
• NeISF: Neural Incident Stokes Field for Geometry and Material Estimation [50.588983686271284]
  Multi-view inverse rendering is the problem of estimating the scene parameters such as shapes, materials, or illuminations from a sequence of images captured under different viewpoints. We propose Neural Incident Stokes Fields (NeISF), a multi-view inverse framework that reduces ambiguities using polarization cues.
  arXiv  Detail & Related papers  (2023-11-22T06:28:30Z)
• Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering [84.37776381343662]
  Mip-NeRF proposes a multiscale representation as a conical frustum to encode scale information. We propose mip voxel grids (Mip-VoG), an explicit multiscale representation for real-time anti-aliasing rendering. Our approach is the first to offer multiscale training and real-time anti-aliasing rendering simultaneously.
  arXiv  Detail & Related papers  (2023-04-20T04:05:22Z)
• A Geometric Model for Polarization Imaging on Projective Cameras [5.381004207943598]
  We present a geometric model describing how a general projective camera captures the light polarization state. Our model is implemented as a pre-processing operation acting on raw images, followed by a per-pixel rotation of the reconstructed normal field. Experiments on existing and new datasets demonstrate the accuracy of the model when applied to commercially available polarimetric cameras.
  arXiv  Detail & Related papers  (2022-11-29T17:12:26Z)
• HSurf-Net: Normal Estimation for 3D Point Clouds by Learning Hyper Surfaces [54.77683371400133]
  We propose a novel normal estimation method called HSurf-Net, which can accurately predict normals from point clouds with noise and density variations. Experimental results show that our HSurf-Net achieves the state-of-the-art performance on the synthetic shape dataset.
  arXiv  Detail & Related papers  (2022-10-13T16:39:53Z)
• Deep Photometric Stereo for Non-Lambertian Surfaces [89.05501463107673]
  We introduce a fully convolutional deep network for calibrated photometric stereo, which we call PS-FCN. PS-FCN learns the mapping from reflectance observations to surface normal, and is able to handle surfaces with general and unknown isotropic reflectance. To deal with the uncalibrated scenario where light directions are unknown, we introduce a new convolutional network, named LCNet, to estimate light directions from input images.
  arXiv  Detail & Related papers  (2020-07-26T15:20:53Z)
• Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization [99.96632216070718]
  Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution. The high-dimensional spatiality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. We propose a novel learning-based LF framework, in which each view of an LF image is first individually super-resolved.
  arXiv  Detail & Related papers  (2020-04-05T14:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.