Multi-resolution Monocular Depth Map Fusion by Self-supervised Gradient-based Composition

• URL: http://arxiv.org/abs/2212.01538v1
• Date: Sat, 3 Dec 2022 05:13:50 GMT
• Title: Multi-resolution Monocular Depth Map Fusion by Self-supervised Gradient-based Composition
• Authors: Yaqiao Dai, Renjiao Yi, Chenyang Zhu, Hongjun He and Kai Xu
• Abstract summary: We propose a novel depth map fusion module to combine the advantages of estimations with multi-resolution inputs. Our lightweight depth fusion is one-shot and runs in real-time, making our method 80X faster than a state-of-the-art depth fusion method.
• Score: 14.246972408737987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monocular depth estimation is a challenging problem on which deep neural networks have demonstrated great potential. However, depth maps predicted by existing deep models usually lack fine-grained details due to the convolution operations and the down-samplings in networks. We find that increasing input resolution is helpful to preserve more local details while the estimation at low resolution is more accurate globally. Therefore, we propose a novel depth map fusion module to combine the advantages of estimations with multi-resolution inputs. Instead of merging the low- and high-resolution estimations equally, we adopt the core idea of Poisson fusion, trying to implant the gradient domain of high-resolution depth into the low-resolution depth. While classic Poisson fusion requires a fusion mask as supervision, we propose a self-supervised framework based on guided image filtering. We demonstrate that this gradient-based composition performs much better at noisy immunity, compared with the state-of-the-art depth map fusion method. Our lightweight depth fusion is one-shot and runs in real-time, making our method 80X faster than a state-of-the-art depth fusion method. Quantitative evaluations demonstrate that the proposed method can be integrated into many fully convolutional monocular depth estimation backbones with a significant performance boost, leading to state-of-the-art results of detail enhancement on depth maps.

Related papers

• Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios [103.72094710263656]
  This paper presents a novel approach that identifies and integrates dominant cross-modality depth features with a learning-based framework. We propose a novel confidence loss steering a confidence predictor network to yield a confidence map specifying latent potential depth areas. With the resulting confidence map, we propose a multi-modal fusion network that fuses the final depth in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-02-19T04:39:16Z)
• V-FUSE: Volumetric Depth Map Fusion with Long-Range Constraints [6.7197802356130465]
  We introduce a learning-based depth map fusion framework that accepts a set of depth and confidence maps generated by a Multi-View Stereo (MVS) algorithm as input and improves them. We also introduce a depth search window estimation sub-network trained jointly with the larger fusion sub-network to reduce the depth hypothesis search space along each ray. Our method learns to model depth consensus and violations of visibility constraints directly from the data.
  arXiv  Detail & Related papers  (2023-08-17T00:39:56Z)
• Improving Monocular Visual Odometry Using Learned Depth [84.05081552443693]
  We propose a framework to exploit monocular depth estimation for improving visual odometry (VO) The core of our framework is a monocular depth estimation module with a strong generalization capability for diverse scenes. Compared with current learning-based VO methods, our method demonstrates a stronger generalization ability to diverse scenes.
  arXiv  Detail & Related papers  (2022-04-04T06:26:46Z)
• Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging [14.279471205248534]
  We show how a consistent scene structure and high-frequency details affect depth estimation performance. We present a double estimation method that improves the whole-image depth estimation and a patch selection method that adds local details. We demonstrate that by merging estimations at different resolutions with changing context, we can generate multi-megapixel depth maps with a high level of detail.
  arXiv  Detail & Related papers  (2021-05-28T17:55:15Z)
• Towards Unpaired Depth Enhancement and Super-Resolution in the Wild [121.96527719530305]
  State-of-the-art data-driven methods of depth map super-resolution rely on registered pairs of low- and high-resolution depth maps of the same scenes. We consider an approach to depth map enhancement based on learning from unpaired data.
  arXiv  Detail & Related papers  (2021-05-25T16:19:16Z)
• FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Monocular Depth Completion [15.01291779855834]
  Recent approaches mainly formulate the depth completion as a one-stage end-to-end learning task. We propose a novel end-to-end residual learning framework, which formulates the depth completion as a two-stage learning task.
  arXiv  Detail & Related papers  (2020-12-15T13:09:56Z)
• HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation [14.81943833870932]
  We present an improvedDepthNet, HR-Depth, with two effective strategies. Using Resnet-18 as the encoder, HR-Depth surpasses all pre-vious state-of-the-art(SoTA) methods with the least param-eters at both high and low resolution.
  arXiv  Detail & Related papers  (2020-12-14T09:15:15Z)
• Efficient Depth Completion Using Learned Bases [94.0808155168311]
  We propose a new global geometry constraint for depth completion. By assuming depth maps often lay on low dimensional subspaces, a dense depth map can be approximated by a weighted sum of full-resolution principal depth bases.
  arXiv  Detail & Related papers  (2020-12-02T11:57:37Z)
• Balanced Depth Completion between Dense Depth Inference and Sparse Range Measurements via KISS-GP [14.158132769768578]
  Estimating a dense and accurate depth map is the key requirement for autonomous driving and robotics. Recent advances in deep learning have allowed depth estimation in full resolution from a single image. Despite this impressive result, many deep-learning-based monocular depth estimation algorithms have failed to keep their accuracy yielding a meter-level estimation error.
  arXiv  Detail & Related papers  (2020-08-12T08:07:55Z)
• RoutedFusion: Learning Real-time Depth Map Fusion [73.0378509030908]
  We present a novel real-time capable machine learning-based method for depth map fusion. We propose a neural network that predicts non-linear updates to better account for typical fusion errors. Our network is composed of a 2D depth routing network and a 3D depth fusion network which efficiently handle sensor-specific noise and outliers.
  arXiv  Detail & Related papers  (2020-01-13T16:46:41Z)
• Video Depth Estimation by Fusing Flow-to-Depth Proposals [65.24533384679657]
  We present an approach with a differentiable flow-to-depth layer for video depth estimation. The model consists of a flow-to-depth layer, a camera pose refinement module, and a depth fusion network. Our approach outperforms state-of-the-art depth estimation methods, and has reasonable cross dataset generalization capability.
  arXiv  Detail & Related papers  (2019-12-30T10:45:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.