Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo

• URL: http://arxiv.org/abs/2011.13117v2
• Date: Wed, 26 May 2021 00:09:24 GMT
• Title: Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo
• Authors: Seung-Hwan Baek, Felix Heide
• Abstract summary: We introduce a novel differentiable image formation model for active stereo, relying on both wave and geometric optics, and a novel trinocular reconstruction network. The jointly optimized pattern, which we dub "Polka Lines," together with the reconstruction network, achieve state-of-the-art active-stereo depth estimates across imaging conditions.
• Score: 52.68109922159688
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Active stereo cameras that recover depth from structured light captures have become a cornerstone sensor modality for 3D scene reconstruction and understanding tasks across application domains. Existing active stereo cameras project a pseudo-random dot pattern on object surfaces to extract disparity independently of object texture. Such hand-crafted patterns are designed in isolation from the scene statistics, ambient illumination conditions, and the reconstruction method. In this work, we propose the first method to jointly learn structured illumination and reconstruction, parameterized by a diffractive optical element and a neural network, in an end-to-end fashion. To this end, we introduce a novel differentiable image formation model for active stereo, relying on both wave and geometric optics, and a novel trinocular reconstruction network. The jointly optimized pattern, which we dub "Polka Lines," together with the reconstruction network, achieve state-of-the-art active-stereo depth estimates across imaging conditions. We validate the proposed method in simulation and on a hardware prototype, and show that our method outperforms existing active stereo systems.

Related papers

• Neural Active Structure-from-Motion in Dark and Textureless Environment [1.3378968838200862]
  We propose a simultaneous shape reconstruction and pose estimation technique for structured light (SL) systems. The proposed method is able to achieve accurate shape reconstruction as well as pose estimation from images where only projected patterns are observed.
  arXiv  Detail & Related papers  (2024-10-20T12:29:01Z)
• ActiveNeuS: Neural Signed Distance Fields for Active Stereo [1.2470540422791463]
  We propose Neural Signed Distance Field for active stereo systems to enable implicit correspondence search and triangulation in generalized Structured Light. With our technique, textureless or equivalent surfaces by low light condition are successfully reconstructed even with a small number of captured images. Experiments were conducted to confirm that the proposed method could achieve state-of-the-art reconstruction quality under such severe condition.
  arXiv  Detail & Related papers  (2024-10-20T12:20:47Z)
• Learning Photometric Feature Transform for Free-form Object Scan [34.61673205691415]
  We propose a novel framework to automatically learn to aggregate and transform photometric measurements from unstructured views. We build a system to reconstruct the geometry and anisotropic reflectance of a variety of challenging objects from hand-held scans. Results are validated against reconstructions from a professional 3D scanner and photographs, and compare favorably with state-of-the-art techniques.
  arXiv  Detail & Related papers  (2023-08-07T11:34:27Z)
• TMO: Textured Mesh Acquisition of Objects with a Mobile Device by using Differentiable Rendering [54.35405028643051]
  We present a new pipeline for acquiring a textured mesh in the wild with a single smartphone. Our method first introduces an RGBD-aided structure from motion, which can yield filtered depth maps. We adopt the neural implicit surface reconstruction method, which allows for high-quality mesh.
  arXiv  Detail & Related papers  (2023-03-27T10:07:52Z)
• Deep Learning Methods for Calibrated Photometric Stereo and Beyond [86.57469194387264]
  Photometric stereo recovers the surface normals of an object from multiple images with varying shading cues. Deep learning methods have shown a powerful ability in the context of photometric stereo against non-Lambertian surfaces.
  arXiv  Detail & Related papers  (2022-12-16T11:27:44Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo [103.08512487830669]
  We present a modern solution to the multi-view photometric stereo problem (MVPS) We procure the surface orientation using a photometric stereo (PS) image formation model and blend it with a multi-view neural radiance field representation to recover the object's surface geometry. Our method performs neural rendering of multi-view images while utilizing surface normals estimated by a deep photometric stereo network.
  arXiv  Detail & Related papers  (2021-10-11T20:20:03Z)
• Active stereo vision three-dimensional reconstruction by RGB dot pattern projection and ray intersection [11.878820609988695]
  We propose a new pattern extraction method and a new stereo vision matching method based on our novel structured light pattern. Experimental results showed that the proposed approach could reconstruct the 3D shape of the object significantly more robustly than state of the art methods.
  arXiv  Detail & Related papers  (2020-03-30T10:13:28Z)
• Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
  We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object. We first estimate per-view depth maps using a deep multi-view stereo network. These depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture.
  arXiv  Detail & Related papers  (2020-03-27T21:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.