360$^\circ$ High-Resolution Depth Estimation via Uncertainty-aware Structural Knowledge Transfer

• URL: http://arxiv.org/abs/2304.07967v3
• Date: Thu, 23 May 2024 08:37:24 GMT
• Title: 360$^\circ$ High-Resolution Depth Estimation via Uncertainty-aware Structural Knowledge Transfer
• Authors: Zidong Cao, Hao Ai, Athanasios V. Vasilakos, Lin Wang,
• Abstract summary: To predict high-resolution (HR) omnidirectional depth map, existing methods typically leverage HR omnidirectional image (ODI) as the input via fully-supervised learning. In this paper, we explore for the first time to estimate the HR omnidirectional depth directly from a low-resolution (LR) ODI, when no HR depth GT map is available. Our key idea is to transfer the scene structural knowledge from the HR image modality and the corresponding LR depth maps to achieve the goal of HR depth estimation without any extra inference cost.
• Score: 8.988255747467333
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: To predict high-resolution (HR) omnidirectional depth map, existing methods typically leverage HR omnidirectional image (ODI) as the input via fully-supervised learning. However, in practice, taking HR ODI as input is undesired due to resource-constrained devices. In addition, depth maps are often with lower resolution than color images. Therefore, in this paper, we explore for the first time to estimate the HR omnidirectional depth directly from a low-resolution (LR) ODI, when no HR depth GT map is available. Our key idea is to transfer the scene structural knowledge from the HR image modality and the corresponding LR depth maps to achieve the goal of HR depth estimation without any extra inference cost. Specifically, we introduce ODI super-resolution (SR) as an auxiliary task and train both tasks collaboratively in a weakly supervised manner to boost the performance of HR depth estimation. The ODI SR task extracts the scene structural knowledge via uncertainty estimation. Buttressed by this, a scene structural knowledge transfer (SSKT) module is proposed with two key components. First, we employ a cylindrical implicit interpolation function (CIIF) to learn cylindrical neural interpolation weights for feature up-sampling and share the parameters of CIIFs between the two tasks. Then, we propose a feature distillation (FD) loss that provides extra structural regularization to help the HR depth estimation task learn more scene structural knowledge. Extensive experiments demonstrate that our weakly-supervised method outperforms baseline methods, and even achieves comparable performance with the fully-supervised methods.

Related papers

• Single-Point Supervised High-Resolution Dynamic Network for Infrared Small Target Detection [7.0456782736205685]
  We propose a single-point supervised high-resolution dynamic network (SSHD-Net) It achieves state-of-the-art (SOTA) detection performance using only single-point supervision. Experiments on the publicly available datasets NUDT-SIRST and IRSTD-1k demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-08-04T09:44:47Z)
• Memory-augmented Deep Unfolding Network for Guided Image Super-resolution [67.83489239124557]
  Guided image super-resolution (GISR) aims to obtain a high-resolution (HR) target image by enhancing the spatial resolution of a low-resolution (LR) target image under the guidance of a HR image. Previous model-based methods mainly takes the entire image as a whole, and assume the prior distribution between the HR target image and the HR guidance image. We propose a maximal a posterior (MAP) estimation model for GISR with two types of prior on the HR target image.
  arXiv  Detail & Related papers  (2022-02-12T15:37:13Z)
• Weakly-Supervised Monocular Depth Estimationwith Resolution-Mismatched Data [73.9872931307401]
  We propose a novel weakly-supervised framework to train a monocular depth estimation network. The proposed framework is composed of a sharing weight monocular depth estimation network and a depth reconstruction network for distillation. Experimental results demonstrate that our method achieves superior performance than unsupervised and semi-supervised learning based schemes.
  arXiv  Detail & Related papers  (2021-09-23T18:04:12Z)
• BridgeNet: A Joint Learning Network of Depth Map Super-Resolution and Monocular Depth Estimation [60.34562823470874]
  We propose a joint learning network of depth map super-resolution (DSR) and monocular depth estimation (MDE) without introducing additional supervision labels. One is the high-frequency attention bridge (HABdg) designed for the feature encoding process, which learns the high-frequency information of the MDE task to guide the DSR task. The other is the content guidance bridge (CGBdg) designed for the depth map reconstruction process, which provides the content guidance learned from DSR task for MDE task.
  arXiv  Detail & Related papers  (2021-07-27T01:28:23Z)
• Towards Fast and Accurate Real-World Depth Super-Resolution: Benchmark Dataset and Baseline [48.69396457721544]
  We build a large-scale dataset named "RGB-D-D" to promote the study of depth map super-resolution (SR) We provide a fast depth map super-resolution (FDSR) baseline, in which the high-frequency component adaptively decomposed from RGB image to guide the depth map SR. For the real-world LR depth maps, our algorithm can produce more accurate HR depth maps with clearer boundaries and to some extent correct the depth value errors.
  arXiv  Detail & Related papers  (2021-04-13T13:27:26Z)
• High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion [84.24973877109181]
  We propose a novel attention-based hierarchical multi-modal fusion network for guided DSR. We show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.
  arXiv  Detail & Related papers  (2021-04-04T03:28:33Z)
• Learning Scene Structure Guidance via Cross-Task Knowledge Transfer for Single Depth Super-Resolution [35.21324004883027]
  Existing color-guided depth super-resolution (DSR) approaches require paired RGB-D data as training samples where the RGB image is used as structural guidance to recover the degraded depth map due to their geometrical similarity. We explore for the first time to learn the cross-modality knowledge at training stage, where both RGB and depth modalities are available, but test on the target dataset, where only single depth modality exists. Specifically, we construct an auxiliary depth estimation (DE) task that takes an RGB image as input to estimate a depth map, and train both DSR task and DE task collaboratively to boost the performance of
  arXiv  Detail & Related papers  (2021-03-24T03:08:25Z)
• Channel Attention based Iterative Residual Learning for Depth Map Super-Resolution [58.626803922196146]
  We argue that DSR models trained on synthetic dataset are restrictive and not effective in dealing with real-world DSR tasks. We make two contributions in tackling real-world degradation of different depth sensors. We propose a new framework for real-world DSR, which consists of four modules.
  arXiv  Detail & Related papers  (2020-06-02T09:12:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.