Towards Multimodal Depth Estimation from Light Fields

• URL: http://arxiv.org/abs/2203.16542v2
• Date: Fri, 1 Apr 2022 10:55:33 GMT
• Title: Towards Multimodal Depth Estimation from Light Fields
• Authors: Titus Leistner, Radek Mackowiak, Lynton Ardizzone, Ullrich K\"othe, Carsten Rother
• Abstract summary: Current depth estimation methods only consider a single "true" depth, even when multiple objects at different depths contributed to the color of a single pixel. We argue that this is due current methods only considering a single "true" depth, even when multiple objects at different depths contributed to the color of a single pixel. We contribute the first "multimodal light field depth dataset" that contains the depths of all objects which contribute to the color of a pixel.
• Score: 29.26003765978794
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Light field applications, especially light field rendering and depth estimation, developed rapidly in recent years. While state-of-the-art light field rendering methods handle semi-transparent and reflective objects well, depth estimation methods either ignore these cases altogether or only deliver a weak performance. We argue that this is due current methods only considering a single "true" depth, even when multiple objects at different depths contributed to the color of a single pixel. Based on the simple idea of outputting a posterior depth distribution instead of only a single estimate, we develop and explore several different deep-learning-based approaches to the problem. Additionally, we contribute the first "multimodal light field depth dataset" that contains the depths of all objects which contribute to the color of a pixel. This allows us to supervise the multimodal depth prediction and also validate all methods by measuring the KL divergence of the predicted posteriors. With our thorough analysis and novel dataset, we aim to start a new line of depth estimation research that overcomes some of the long-standing limitations of this field.

Related papers

• Transparent Object Depth Completion [11.825680661429825]
  The perception of transparent objects for grasp and manipulation remains a major challenge. Existing robotic grasp methods which heavily rely on depth maps are not suitable for transparent objects due to their unique visual properties. We propose an end-to-end network for transparent object depth completion that combines the strengths of single-view RGB-D based depth completion and multi-view depth estimation.
  arXiv  Detail & Related papers  (2024-05-24T07:38:06Z)
• Robust Depth Enhancement via Polarization Prompt Fusion Tuning [112.88371907047396]
  We present a framework that leverages polarization imaging to improve inaccurate depth measurements from various depth sensors. Our method first adopts a learning-based strategy where a neural network is trained to estimate a dense and complete depth map from polarization data and a sensor depth map from different sensors. To further improve the performance, we propose a Polarization Prompt Fusion Tuning (PPFT) strategy to effectively utilize RGB-based models pre-trained on large-scale datasets.
  arXiv  Detail & Related papers  (2024-04-05T17:55:33Z)
• Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios [103.72094710263656]
  This paper presents a novel approach that identifies and integrates dominant cross-modality depth features with a learning-based framework. We propose a novel confidence loss steering a confidence predictor network to yield a confidence map specifying latent potential depth areas. With the resulting confidence map, we propose a multi-modal fusion network that fuses the final depth in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-02-19T04:39:16Z)
• Depth Insight -- Contribution of Different Features to Indoor Single-image Depth Estimation [8.712751056826283]
  We quantify the relative contributions of the known cues of depth in a monocular depth estimation setting. Our work uses feature extraction techniques to relate the single features of shape, texture, colour and saturation, taken in isolation, to predict depth.
  arXiv  Detail & Related papers  (2023-11-16T17:38:21Z)
• Blur aware metric depth estimation with multi-focus plenoptic cameras [8.508198765617196]
  We present a new metric depth estimation algorithm using only raw images from a multi-focus plenoptic camera. The proposed approach is especially suited for the multi-focus configuration where several micro-lenses with different focal lengths are used.
  arXiv  Detail & Related papers  (2023-08-08T13:38:50Z)
• Probabilistic and Geometric Depth: Detecting Objects in Perspective [78.00922683083776]
  3D object detection is an important capability needed in various practical applications such as driver assistance systems. Monocular 3D detection, as an economical solution compared to conventional settings relying on binocular vision or LiDAR, has drawn increasing attention recently but still yields unsatisfactory results. This paper first presents a systematic study on this problem and observes that the current monocular 3D detection problem can be simplified as an instance depth estimation problem.
  arXiv  Detail & Related papers  (2021-07-29T16:30:33Z)
• Learning Multi-modal Information for Robust Light Field Depth Estimation [32.64928379844675]
  Existing learning-based depth estimation methods from the focal stack lead to suboptimal performance because of the defocus blur. We propose a multi-modal learning method for robust light field depth estimation. Our method achieves superior performance than existing representative methods on two light field datasets.
  arXiv  Detail & Related papers  (2021-04-13T06:51:27Z)
• Efficient Depth Completion Using Learned Bases [94.0808155168311]
  We propose a new global geometry constraint for depth completion. By assuming depth maps often lay on low dimensional subspaces, a dense depth map can be approximated by a weighted sum of full-resolution principal depth bases.
  arXiv  Detail & Related papers  (2020-12-02T11:57:37Z)
• View-consistent 4D Light Field Depth Estimation [37.04038603184669]
  We propose a method to compute depth maps for every sub-aperture image in a light field in a view consistent way. Our method precisely defines depth edges via EPIs, then we diffuse these edges spatially within the central view.
  arXiv  Detail & Related papers  (2020-09-09T01:47:34Z)
• Occlusion-Aware Depth Estimation with Adaptive Normal Constraints [85.44842683936471]
  We present a new learning-based method for multi-frame depth estimation from a color video. Our method outperforms the state-of-the-art in terms of depth estimation accuracy.
  arXiv  Detail & Related papers  (2020-04-02T07:10:45Z)
• Single Image Depth Estimation Trained via Depth from Defocus Cues [105.67073923825842]
  Estimating depth from a single RGB image is a fundamental task in computer vision. In this work, we rely, instead of different views, on depth from focus cues. We present results that are on par with supervised methods on KITTI and Make3D datasets and outperform unsupervised learning approaches.
  arXiv  Detail & Related papers  (2020-01-14T20:22:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.