FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Monocular Depth Completion

• URL: http://arxiv.org/abs/2012.08270v1
• Date: Tue, 15 Dec 2020 13:09:56 GMT
• Title: FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Monocular Depth Completion
• Authors: Lina Liu, Xibin Song, Xiaoyang Lyu, Junwei Diao, Mengmeng Wang, Yong Liu, Liangjun Zhang
• Abstract summary: Recent approaches mainly formulate the depth completion as a one-stage end-to-end learning task. We propose a novel end-to-end residual learning framework, which formulates the depth completion as a two-stage learning task.
• Score: 15.01291779855834
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Depth completion aims to recover a dense depth map from a sparse depth map with the corresponding color image as input. Recent approaches mainly formulate the depth completion as a one-stage end-to-end learning task, which outputs dense depth maps directly. However, the feature extraction and supervision in one-stage frameworks are insufficient, limiting the performance of these approaches. To address this problem, we propose a novel end-to-end residual learning framework, which formulates the depth completion as a two-stage learning task, i.e., a sparse-to-coarse stage and a coarse-to-fine stage. First, a coarse dense depth map is obtained by a simple CNN framework. Then, a refined depth map is further obtained using a residual learning strategy in the coarse-to-fine stage with coarse depth map and color image as input. Specially, in the coarse-to-fine stage, a channel shuffle extraction operation is utilized to extract more representative features from color image and coarse depth map, and an energy based fusion operation is exploited to effectively fuse these features obtained by channel shuffle operation, thus leading to more accurate and refined depth maps. We achieve SoTA performance in RMSE on KITTI benchmark. Extensive experiments on other datasets future demonstrate the superiority of our approach over current state-of-the-art depth completion approaches.

Related papers

• DepthSplat: Connecting Gaussian Splatting and Depth [90.06180236292866]
  We present DepthSplat to connect Gaussian splatting and depth estimation. We first contribute a robust multi-view depth model by leveraging pre-trained monocular depth features. We also show that Gaussian splatting can serve as an unsupervised pre-training objective.
  arXiv  Detail & Related papers  (2024-10-17T17:59:58Z)
• SteeredMarigold: Steering Diffusion Towards Depth Completion of Largely Incomplete Depth Maps [3.399289369740637]
  SteeredMarigold is a training-free, zero-shot depth completion method. It produces metric dense depth even for largely incomplete depth maps. Our code will be publicly available.
  arXiv  Detail & Related papers  (2024-09-16T11:52:13Z)
• Bilateral Propagation Network for Depth Completion [41.163328523175466]
  Depth completion aims to derive a dense depth map from sparse depth measurements with a synchronized color image. Current state-of-the-art (SOTA) methods are predominantly propagation-based, which work as an iterative refinement on the initial estimated dense depth. We present a Bilateral Propagation Network (BP-Net), that propagates depth at the earliest stage to avoid directly convolving on sparse data.
  arXiv  Detail & Related papers  (2024-03-17T16:48:46Z)
• Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios [103.72094710263656]
  This paper presents a novel approach that identifies and integrates dominant cross-modality depth features with a learning-based framework. We propose a novel confidence loss steering a confidence predictor network to yield a confidence map specifying latent potential depth areas. With the resulting confidence map, we propose a multi-modal fusion network that fuses the final depth in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-02-19T04:39:16Z)
• LRRU: Long-short Range Recurrent Updating Networks for Depth Completion [45.48580252300282]
  Long-short Range Recurrent Updating (LRRU) network is proposed to accomplish depth completion more efficiently. LRRU first roughly fills the sparse input to obtain an initial dense depth map, and then iteratively updates it through learned spatially-variant kernels. Our initial depth map has coarse but complete scene depth information, which helps relieve the burden of directly regressing the dense depth from sparse ones.
  arXiv  Detail & Related papers  (2023-10-13T09:04:52Z)
• RigNet: Repetitive Image Guided Network for Depth Completion [20.66405067066299]
  Recent approaches mainly focus on image guided learning to predict dense results. blurry image guidance and object structures in depth still impede the performance of image guided frameworks. We explore a repetitive design in our image guided network to sufficiently and gradually recover depth values. Our method achieves state-of-the-art result on the NYUv2 dataset and ranks 1st on the KITTI benchmark at the time of submission.
  arXiv  Detail & Related papers  (2021-07-29T08:00:33Z)
• Depth Completion using Plane-Residual Representation [84.63079529738924]
  We introduce a novel way of interpreting depth information with the closest depth plane label $p$ and a residual value $r$, as we call it, Plane-Residual (PR) representation. By interpreting depth information in PR representation and using our corresponding depth completion network, we were able to acquire improved depth completion performance with faster computation.
  arXiv  Detail & Related papers  (2021-04-15T10:17:53Z)
• Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion [56.85837052421469]
  Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars. In this paper, we study the problem of predicting dense depth from a single RGB image with optional sparse measurements from low-cost active depth sensors. We introduce Sparse Networks (SANs), a new module enabling monodepth networks to perform both the tasks of depth prediction and completion.
  arXiv  Detail & Related papers  (2021-03-30T21:22:26Z)
• Efficient Depth Completion Using Learned Bases [94.0808155168311]
  We propose a new global geometry constraint for depth completion. By assuming depth maps often lay on low dimensional subspaces, a dense depth map can be approximated by a weighted sum of full-resolution principal depth bases.
  arXiv  Detail & Related papers  (2020-12-02T11:57:37Z)
• Depth Completion Using a View-constrained Deep Prior [73.21559000917554]
  Recent work has shown that the structure of convolutional neural networks (CNNs) induces a strong prior that favors natural images. This prior, known as a deep image prior (DIP), is an effective regularizer in inverse problems such as image denoising and inpainting. We extend the concept of the DIP to depth images. Given color images and noisy and incomplete target depth maps, we reconstruct a depth map restored by virtue of using the CNN network structure as a prior.
  arXiv  Detail & Related papers  (2020-01-21T21:56:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.