Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration

• URL: http://arxiv.org/abs/2012.00301v1
• Date: Tue, 1 Dec 2020 06:53:57 GMT
• Title: Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration
• Authors: Liyuan Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Hongguang Zhang, and Hongdong Li
• Abstract summary: We study the formation of the DP pair which links the blur and the depth information. We propose an end-to-end DDDNet (DP-based Depth and De Network) to jointly estimate the depth and restore the image.
• Score: 77.1056200937214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The dual-pixel (DP) hardware works by splitting each pixel in half and creating an image pair in a single snapshot. Several works estimate depth/inverse depth by treating the DP pair as a stereo pair. However, dual-pixel disparity only occurs in image regions with the defocus blur. The heavy defocus blur in DP pairs affects the performance of matching-based depth estimation approaches. Instead of removing the blur effect blindly, we study the formation of the DP pair which links the blur and the depth information. In this paper, we propose a mathematical DP model which can benefit depth estimation by the blur. These explorations motivate us to propose an end-to-end DDDNet (DP-based Depth and Deblur Network) to jointly estimate the depth and restore the image. Moreover, we define a reblur loss, which reflects the relationship of the DP image formation process with depth information, to regularise our depth estimate in training. To meet the requirement of a large amount of data for learning, we propose the first DP image simulator which allows us to create datasets with DP pairs from any existing RGBD dataset. As a side contribution, we collect a real dataset for further research. Extensive experimental evaluation on both synthetic and real datasets shows that our approach achieves competitive performance compared to state-of-the-art approaches.

Related papers

• Single Image Depth Prediction Made Better: A Multivariate Gaussian Take [163.14849753700682]
  We introduce an approach that performs continuous modeling of per-pixel depth. Our method's accuracy (named MG) is among the top on the KITTI depth-prediction benchmark leaderboard.
  arXiv  Detail & Related papers  (2023-03-31T16:01:03Z)
• RA-Depth: Resolution Adaptive Self-Supervised Monocular Depth Estimation [27.679479140943503]
  We propose a resolution adaptive self-supervised monocular depth estimation method (RA-Depth) by learning the scale invariance of the scene depth. RA-Depth achieves state-of-the-art performance, and also exhibits a good ability of resolution adaptation.
  arXiv  Detail & Related papers  (2022-07-25T08:49:59Z)
• Unsupervised Visible-light Images Guided Cross-Spectrum Depth Estimation from Dual-Modality Cameras [33.77748026254935]
  Cross-spectrum depth estimation aims to provide a depth map in all illumination conditions with a pair of dual-spectrum images. In this paper, we propose an unsupervised visible-light image guided cross-spectrum (i.e., thermal and visible-light, TIR-VIS in short) depth estimation framework. Our method achieves better performance than the compared existing methods.
  arXiv  Detail & Related papers  (2022-04-30T12:58:35Z)
• Facial Depth and Normal Estimation using Single Dual-Pixel Camera [81.02680586859105]
  We introduce a DP-oriented Depth/Normal network that reconstructs the 3D facial geometry. It contains the corresponding ground-truth 3D models including depth map and surface normal in metric scale. It achieves state-of-the-art performances over recent DP-based depth/normal estimation methods.
  arXiv  Detail & Related papers  (2021-11-25T05:59:27Z)
• Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning [48.063176079878055]
  We propose a single-image deblurring network that incorporates the two sub-aperture views into a multi-task framework. Our experiments show this multi-task strategy achieves +1dB PSNR improvement over state-of-the-art defocus deblurring methods. These high-quality DP views can be used for other DP-based applications, such as reflection removal.
  arXiv  Detail & Related papers  (2021-08-11T14:45:15Z)
• PDC: Piecewise Depth Completion utilizing Superpixels [0.0]
  Current approaches often rely on CNN-based methods with several known drawbacks. We propose our novel Piecewise Depth Completion (PDC), which works completely without deep learning. In our evaluation, we can show both the influence of the individual proposed processing steps and the overall performance of our method on the challenging KITTI dataset.
  arXiv  Detail & Related papers  (2021-07-14T13:58:39Z)
• Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion [56.85837052421469]
  Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars. In this paper, we study the problem of predicting dense depth from a single RGB image with optional sparse measurements from low-cost active depth sensors. We introduce Sparse Networks (SANs), a new module enabling monodepth networks to perform both the tasks of depth prediction and completion.
  arXiv  Detail & Related papers  (2021-03-30T21:22:26Z)
• Pixel-Pair Occlusion Relationship Map(P2ORM): Formulation, Inference & Application [20.63938300312815]
  We formalize concepts around geometric occlusion in 2D images (i.e., ignoring semantics) We propose a novel unified formulation of both occlusion boundaries and occlusion orientations via a pixel-pair occlusion relation. Experiments on a variety of datasets demonstrate that our method outperforms existing ones on this task. We also propose a new depth map refinement method that consistently improve the performance of state-of-the-art monocular depth estimation methods.
  arXiv  Detail & Related papers  (2020-07-23T15:52:09Z)
• Single Image Depth Estimation Trained via Depth from Defocus Cues [105.67073923825842]
  Estimating depth from a single RGB image is a fundamental task in computer vision. In this work, we rely, instead of different views, on depth from focus cues. We present results that are on par with supervised methods on KITTI and Make3D datasets and outperform unsupervised learning approaches.
  arXiv  Detail & Related papers  (2020-01-14T20:22:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.