Fully Self-Supervised Depth Estimation from Defocus Clue

• URL: http://arxiv.org/abs/2303.10752v4
• Date: Mon, 27 Mar 2023 04:26:50 GMT
• Title: Fully Self-Supervised Depth Estimation from Defocus Clue
• Authors: Haozhe Si, Bin Zhao, Dong Wang, Yunpeng Gao, Mulin Chen, Zhigang Wang, Xuelong Li
• Abstract summary: We propose a self-supervised framework that estimates depth purely from a sparse focal stack. We show that our framework circumvents the needs for the depth and AIF image ground-truth, and receives superior predictions.
• Score: 79.63579768496159
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Depth-from-defocus (DFD), modeling the relationship between depth and defocus pattern in images, has demonstrated promising performance in depth estimation. Recently, several self-supervised works try to overcome the difficulties in acquiring accurate depth ground-truth. However, they depend on the all-in-focus (AIF) images, which cannot be captured in real-world scenarios. Such limitation discourages the applications of DFD methods. To tackle this issue, we propose a completely self-supervised framework that estimates depth purely from a sparse focal stack. We show that our framework circumvents the needs for the depth and AIF image ground-truth, and receives superior predictions, thus closing the gap between the theoretical success of DFD works and their applications in the real world. In particular, we propose (i) a more realistic setting for DFD tasks, where no depth or AIF image ground-truth is available; (ii) a novel self-supervision framework that provides reliable predictions of depth and AIF image under the challenging setting. The proposed framework uses a neural model to predict the depth and AIF image, and utilizes an optical model to validate and refine the prediction. We verify our framework on three benchmark datasets with rendered focal stacks and real focal stacks. Qualitative and quantitative evaluations show that our method provides a strong baseline for self-supervised DFD tasks.

Related papers

• Depth Estimation Based on 3D Gaussian Splatting Siamese Defocus [14.354405484663285]
  We propose a self-supervised framework based on 3D Gaussian splatting and Siamese networks for depth estimation in 3D geometry. The proposed framework has been validated on both artificially synthesized and real blurred datasets.
  arXiv  Detail & Related papers  (2024-09-18T21:36:37Z)
• Self-Supervised Spatially Variant PSF Estimation for Aberration-Aware Depth-from-Defocus [14.383129822833155]
  We propose a novel self-supervised learning method for aberration-aware depth-from-defocus (DfD) In our PSF estimation, we assume rotationally symmetric PSFs and introduce the polar coordinate system. We also handle the focus breathing phenomenon that occurs in real DfD situations.
  arXiv  Detail & Related papers  (2024-02-28T09:07:26Z)
• Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios [103.72094710263656]
  This paper presents a novel approach that identifies and integrates dominant cross-modality depth features with a learning-based framework. We propose a novel confidence loss steering a confidence predictor network to yield a confidence map specifying latent potential depth areas. With the resulting confidence map, we propose a multi-modal fusion network that fuses the final depth in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-02-19T04:39:16Z)
• Depth and DOF Cues Make A Better Defocus Blur Detector [27.33757097343283]
  Defocus blur detection (DBD) separates in-focus and out-of-focus regions in an image. Previous approaches mistakenly mistook homogeneous areas in focus for defocus blur regions. We propose an approach called D-DFFNet, which incorporates depth and DOF cues in an implicit manner.
  arXiv  Detail & Related papers  (2023-06-20T07:03:37Z)
• Single Image Depth Prediction Made Better: A Multivariate Gaussian Take [163.14849753700682]
  We introduce an approach that performs continuous modeling of per-pixel depth. Our method's accuracy (named MG) is among the top on the KITTI depth-prediction benchmark leaderboard.
  arXiv  Detail & Related papers  (2023-03-31T16:01:03Z)
• SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes [58.89295356901823]
  Self-supervised monocular depth estimation has shown impressive results in static scenes. It relies on the multi-view consistency assumption for training networks, however, that is violated in dynamic object regions. We introduce an external pretrained monocular depth estimation model for generating single-image depth prior. Our model can predict sharp and accurate depth maps, even when training from monocular videos of highly-dynamic scenes.
  arXiv  Detail & Related papers  (2022-11-07T16:17:47Z)
• Bridging Unsupervised and Supervised Depth from Focus via All-in-Focus Supervision [10.547816678110417]
  The proposed method can be trained either supervisedly with ground truth depth, or emphunsupervisedly with AiF images as supervisory signals. We show in various experiments that our method outperforms the state-of-the-art methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-08-24T17:09:13Z)
• Variational Monocular Depth Estimation for Reliability Prediction [12.951621755732544]
  Self-supervised learning for monocular depth estimation is widely investigated as an alternative to supervised learning approach. Previous works have successfully improved the accuracy of depth estimation by modifying the model structure. In this paper, we theoretically formulate a variational model for the monocular depth estimation to predict the reliability of the estimated depth image.
  arXiv  Detail & Related papers  (2020-11-24T06:23:51Z)
• Defocus Blur Detection via Depth Distillation [64.78779830554731]
  We introduce depth information into DBD for the first time. In detail, we learn the defocus blur from ground truth and the depth distilled from a well-trained depth estimation network. Our approach outperforms 11 other state-of-the-art methods on two popular datasets.
  arXiv  Detail & Related papers  (2020-07-16T04:58:09Z)
• Video Depth Estimation by Fusing Flow-to-Depth Proposals [65.24533384679657]
  We present an approach with a differentiable flow-to-depth layer for video depth estimation. The model consists of a flow-to-depth layer, a camera pose refinement module, and a depth fusion network. Our approach outperforms state-of-the-art depth estimation methods, and has reasonable cross dataset generalization capability.
  arXiv  Detail & Related papers  (2019-12-30T10:45:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.