Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion

• URL: http://arxiv.org/abs/2103.16690v1
• Date: Tue, 30 Mar 2021 21:22:26 GMT
• Title: Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion
• Authors: Vitor Guizilini, Rares Ambrus, Wolfram Burgard, Adrien Gaidon
• Abstract summary: Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars. In this paper, we study the problem of predicting dense depth from a single RGB image with optional sparse measurements from low-cost active depth sensors. We introduce Sparse Networks (SANs), a new module enabling monodepth networks to perform both the tasks of depth prediction and completion.
• Score: 56.85837052421469
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars. In this paper, we study the problem of predicting dense depth from a single RGB image (monodepth) with optional sparse measurements from low-cost active depth sensors. We introduce Sparse Auxiliary Networks (SANs), a new module enabling monodepth networks to perform both the tasks of depth prediction and completion, depending on whether only RGB images or also sparse point clouds are available at inference time. First, we decouple the image and depth map encoding stages using sparse convolutions to process only the valid depth map pixels. Second, we inject this information, when available, into the skip connections of the depth prediction network, augmenting its features. Through extensive experimental analysis on one indoor (NYUv2) and two outdoor (KITTI and DDAD) benchmarks, we demonstrate that our proposed SAN architecture is able to simultaneously learn both tasks, while achieving a new state of the art in depth prediction by a significant margin.

Related papers

• GraphCSPN: Geometry-Aware Depth Completion via Dynamic GCNs [49.55919802779889]
  We propose a Graph Convolution based Spatial Propagation Network (GraphCSPN) as a general approach for depth completion. In this work, we leverage convolution neural networks as well as graph neural networks in a complementary way for geometric representation learning. Our method achieves the state-of-the-art performance, especially when compared in the case of using only a few propagation steps.
  arXiv  Detail & Related papers  (2022-10-19T17:56:03Z)
• Lightweight Monocular Depth Estimation with an Edge Guided Network [34.03711454383413]
  We present a novel lightweight Edge Guided Depth Estimation Network (EGD-Net) In particular, we start out with a lightweight encoder-decoder architecture and embed an edge guidance branch. In order to aggregate the context information and edge attention features, we design a transformer-based feature aggregation module.
  arXiv  Detail & Related papers  (2022-09-29T14:45:47Z)
• RGB-Depth Fusion GAN for Indoor Depth Completion [29.938869342958125]
  In this paper, we design a novel two-branch end-to-end fusion network, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map. In one branch, we propose an RGB-depth fusion GAN to transfer the RGB image to the fine-grained textured depth map. In the other branch, we adopt adaptive fusion modules named W-AdaIN to propagate the features across the two branches.
  arXiv  Detail & Related papers  (2022-03-21T10:26:38Z)
• Sparse Depth Completion with Semantic Mesh Deformation Optimization [4.03103540543081]
  We propose a neural network with post-optimization, which takes an RGB image and sparse depth samples as input and predicts the complete depth map. Our evaluation results outperform the existing work consistently on both indoor and outdoor datasets.
  arXiv  Detail & Related papers  (2021-12-10T13:01:06Z)
• Depth Completion using Plane-Residual Representation [84.63079529738924]
  We introduce a novel way of interpreting depth information with the closest depth plane label $p$ and a residual value $r$, as we call it, Plane-Residual (PR) representation. By interpreting depth information in PR representation and using our corresponding depth completion network, we were able to acquire improved depth completion performance with faster computation.
  arXiv  Detail & Related papers  (2021-04-15T10:17:53Z)
• PLADE-Net: Towards Pixel-Level Accuracy for Self-Supervised Single-View Depth Estimation with Neural Positional Encoding and Distilled Matting Loss [49.66736599668501]
  We propose a self-supervised single-view pixel-level accurate depth estimation network, called PLADE-Net. Our method shows unprecedented accuracy levels, exceeding 95% in terms of the $delta1$ metric on the KITTI dataset.
  arXiv  Detail & Related papers  (2021-03-12T15:54:46Z)
• CodeVIO: Visual-Inertial Odometry with Learned Optimizable Dense Depth [83.77839773394106]
  We present a lightweight, tightly-coupled deep depth network and visual-inertial odometry system. We provide the network with previously marginalized sparse features from VIO to increase the accuracy of initial depth prediction. We show that it can run in real-time with single-thread execution while utilizing GPU acceleration only for the network and code Jacobian.
  arXiv  Detail & Related papers  (2020-12-18T09:42:54Z)
• Accurate RGB-D Salient Object Detection via Collaborative Learning [101.82654054191443]
  RGB-D saliency detection shows impressive ability on some challenge scenarios. We propose a novel collaborative learning framework where edge, depth and saliency are leveraged in a more efficient way.
  arXiv  Detail & Related papers  (2020-07-23T04:33:36Z)
• Guiding Monocular Depth Estimation Using Depth-Attention Volume [38.92495189498365]
  We propose guiding depth estimation to favor planar structures that are ubiquitous especially in indoor environments. Experiments on two popular indoor datasets, NYU-Depth-v2 and ScanNet, show that our method achieves state-of-the-art depth estimation results.
  arXiv  Detail & Related papers  (2020-04-06T15:45:52Z)
• 3dDepthNet: Point Cloud Guided Depth Completion Network for Sparse Depth and Single Color Image [42.13930269841654]
  Our network offers a novel 3D-to-2D coarse-to-fine dual densification design that is both accurate and lightweight. Experiments on the KITTI dataset show our network achieves state-of-art accuracy while being more efficient.
  arXiv  Detail & Related papers  (2020-03-20T10:19:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.