Revisiting Disparity from Dual-Pixel Images: Physics-Informed Lightweight Depth Estimation

• URL: http://arxiv.org/abs/2411.04714v1
• Date: Wed, 06 Nov 2024 09:03:53 GMT
• Title: Revisiting Disparity from Dual-Pixel Images: Physics-Informed Lightweight Depth Estimation
• Authors: Teppei Kurita, Yuhi Kondo, Legong Sun, Takayuki Sasaki, Sho Nitta, Yasuhiro Hashimoto, Yoshinori Muramatsu, Yusuke Moriuchi,
• Abstract summary: We propose a lightweight disparity estimation method based on a completion-based network. By modeling the DP-specific disparity error parametrically and using it for sampling during training, the network acquires the unique properties of DP. As a result, the proposed method achieved state-of-the-art results while reducing the overall system size to 1/5 of that of the conventional method.
• Score: 3.6337378417255177
• License:
• Abstract: In this study, we propose a high-performance disparity (depth) estimation method using dual-pixel (DP) images with few parameters. Conventional end-to-end deep-learning methods have many parameters but do not fully exploit disparity constraints, which limits their performance. Therefore, we propose a lightweight disparity estimation method based on a completion-based network that explicitly constrains disparity and learns the physical and systemic disparity properties of DP. By modeling the DP-specific disparity error parametrically and using it for sampling during training, the network acquires the unique properties of DP and enhances robustness. This learning also allows us to use a common RGB-D dataset for training without a DP dataset, which is labor-intensive to acquire. Furthermore, we propose a non-learning-based refinement framework that efficiently handles inherent disparity expansion errors by appropriately refining the confidence map of the network output. As a result, the proposed method achieved state-of-the-art results while reducing the overall system size to 1/5 of that of the conventional method, even without using the DP dataset for training, thereby demonstrating its effectiveness. The code and dataset are available on our project site.

Related papers

• Learning from Multi-Perception Features for Real-Word Image Super-resolution [87.71135803794519]
  We propose a novel SR method called MPF-Net that leverages multiple perceptual features of input images. Our method incorporates a Multi-Perception Feature Extraction (MPFE) module to extract diverse perceptual information. We also introduce a contrastive regularization term (CR) that improves the model's learning capability.
  arXiv  Detail & Related papers  (2023-05-26T07:35:49Z)
• Single Image Depth Prediction Made Better: A Multivariate Gaussian Take [163.14849753700682]
  We introduce an approach that performs continuous modeling of per-pixel depth. Our method's accuracy (named MG) is among the top on the KITTI depth-prediction benchmark leaderboard.
  arXiv  Detail & Related papers  (2023-03-31T16:01:03Z)
• A Model-data-driven Network Embedding Multidimensional Features for Tomographic SAR Imaging [5.489791364472879]
  We propose a new model-data-driven network to achieve tomoSAR imaging based on multi-dimensional features. We add two 2D processing modules, both convolutional encoder-decoder structures, to enhance multi-dimensional features of the imaging scene effectively. Compared with the conventional CS-based FISTA method and DL-based gamma-Net method, the result of our proposed method has better performance on completeness while having decent imaging accuracy.
  arXiv  Detail & Related papers  (2022-11-28T02:01:43Z)
• Dense Depth Distillation with Out-of-Distribution Simulated Images [30.79756881887895]
  We study data-free knowledge distillation (KD) for monocular depth estimation (MDE) KD learns a lightweight model for real-world depth perception tasks by compressing it from a trained teacher model while lacking training data in the target domain. We show that our method outperforms the baseline KD by a good margin and even slightly better performance with as few as 1/6 of training images.
  arXiv  Detail & Related papers  (2022-08-26T07:10:01Z)
• RA-Depth: Resolution Adaptive Self-Supervised Monocular Depth Estimation [27.679479140943503]
  We propose a resolution adaptive self-supervised monocular depth estimation method (RA-Depth) by learning the scale invariance of the scene depth. RA-Depth achieves state-of-the-art performance, and also exhibits a good ability of resolution adaptation.
  arXiv  Detail & Related papers  (2022-07-25T08:49:59Z)
• Cooperative Deep $Q$-learning Framework for Environments Providing Image Feedback [5.607676459156789]
  We address two key challenges in deep reinforcement learning setting, sample inefficiency and slow learning, with a dual NN-driven learning approach. In particular, we develop a temporal difference (TD) error-driven learning approach, where we introduce a set of linear transformations of the TD error to directly update the parameters of each layer in the deep NN. We show that the proposed methods enables faster learning and convergence and requires reduced buffer size.
  arXiv  Detail & Related papers  (2021-10-28T17:12:41Z)
• An Adaptive Framework for Learning Unsupervised Depth Completion [59.17364202590475]
  We present a method to infer a dense depth map from a color image and associated sparse depth measurements. We show that regularization and co-visibility are related via the fitness of the model to data and can be unified into a single framework.
  arXiv  Detail & Related papers  (2021-06-06T02:27:55Z)
• Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration [77.1056200937214]
  We study the formation of the DP pair which links the blur and the depth information. We propose an end-to-end DDDNet (DP-based Depth and De Network) to jointly estimate the depth and restore the image.
  arXiv  Detail & Related papers  (2020-12-01T06:53:57Z)
• Deep Unfolding Network for Image Super-Resolution [159.50726840791697]
  This paper proposes an end-to-end trainable unfolding network which leverages both learning-based methods and model-based methods. The proposed network inherits the flexibility of model-based methods to super-resolve blurry, noisy images for different scale factors via a single model.
  arXiv  Detail & Related papers  (2020-03-23T17:55:42Z)
• BP-DIP: A Backprojection based Deep Image Prior [49.375539602228415]
  We propose two image restoration approaches: (i) Deep Image Prior (DIP), which trains a convolutional neural network (CNN) from scratch in test time using the degraded image; and (ii) a backprojection (BP) fidelity term, which is an alternative to the standard least squares loss that is usually used in previous DIP works. We demonstrate the performance of the proposed method, termed BP-DIP, on the deblurring task and show its advantages over the plain DIP, with both higher PSNR values and better inference run-time.
  arXiv  Detail & Related papers  (2020-03-11T17:09:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.