Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging

• URL: http://arxiv.org/abs/2402.18102v2
• Date: Sat, 30 Mar 2024 06:06:38 GMT
• Title: Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging
• Authors: Bhargav Ghanekar, Salman Siddique Khan, Pranav Sharma, Shreyas Singh, Vivek Boominathan, Kaushik Mitra, Ashok Veeraraghavan,
• Abstract summary: Single-shot 3D sensing is useful in many application areas such as microscopy, medical imaging, surgical navigation, and autonomous driving. We propose CADS (Coded Aperture Dual-Pixel Sensing), in which we use a coded aperture in the imaging lens along with a DP sensor. Our resulting CADS imaging system demonstrates improvement of >1.5dB PSNR in all-in-focus (AIF) estimates and 5-6% in depth estimation quality over naive DP sensing.
• Score: 25.851398356458425
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Passive, compact, single-shot 3D sensing is useful in many application areas such as microscopy, medical imaging, surgical navigation, and autonomous driving where form factor, time, and power constraints can exist. Obtaining RGB-D scene information over a short imaging distance, in an ultra-compact form factor, and in a passive, snapshot manner is challenging. Dual-pixel (DP) sensors are a potential solution to achieve the same. DP sensors collect light rays from two different halves of the lens in two interleaved pixel arrays, thus capturing two slightly different views of the scene, like a stereo camera system. However, imaging with a DP sensor implies that the defocus blur size is directly proportional to the disparity seen between the views. This creates a trade-off between disparity estimation vs. deblurring accuracy. To improve this trade-off effect, we propose CADS (Coded Aperture Dual-Pixel Sensing), in which we use a coded aperture in the imaging lens along with a DP sensor. In our approach, we jointly learn an optimal coded pattern and the reconstruction algorithm in an end-to-end optimization setting. Our resulting CADS imaging system demonstrates improvement of >1.5dB PSNR in all-in-focus (AIF) estimates and 5-6% in depth estimation quality over naive DP sensing for a wide range of aperture settings. Furthermore, we build the proposed CADS prototypes for DSLR photography settings and in an endoscope and a dermoscope form factor. Our novel coded dual-pixel sensing approach demonstrates accurate RGB-D reconstruction results in simulations and real-world experiments in a passive, snapshot, and compact manner.

Related papers

• Disparity Estimation Using a Quad-Pixel Sensor [12.34044154078824]
  A quad-pixel (QP) sensor is increasingly integrated into commercial mobile cameras. We propose a QP disparity estimation network (QPDNet) We present a synthetic pipeline to generate a training dataset from an existing RGB-Depth dataset.
  arXiv  Detail & Related papers  (2024-09-01T08:50:32Z)
• Continuous Cost Aggregation for Dual-Pixel Disparity Extraction [3.1153758106426603]
  We propose a continuous cost aggregation scheme for Dual-Pixel (DP) images. The proposed algorithm fits parabolas to matching costs and aggregates parabola coefficients along image paths. Experiments on DP data from both DSLR and phone cameras show that the proposed scheme attains state-of-the-art performance in DP disparity estimation.
  arXiv  Detail & Related papers  (2023-06-13T17:26:50Z)
• Learning Dual-Pixel Alignment for Defocus Deblurring [73.80328094662976]
  We propose a Dual-Pixel Alignment Network (DPANet) for defocus deblurring. It is notably superior to state-of-the-art deblurring methods in reducing defocus blur while recovering visually plausible sharp structures and textures.
  arXiv  Detail & Related papers  (2022-04-26T07:02:58Z)
• Facial Depth and Normal Estimation using Single Dual-Pixel Camera [81.02680586859105]
  We introduce a DP-oriented Depth/Normal network that reconstructs the 3D facial geometry. It contains the corresponding ground-truth 3D models including depth map and surface normal in metric scale. It achieves state-of-the-art performances over recent DP-based depth/normal estimation methods.
  arXiv  Detail & Related papers  (2021-11-25T05:59:27Z)
• Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning [48.063176079878055]
  We propose a single-image deblurring network that incorporates the two sub-aperture views into a multi-task framework. Our experiments show this multi-task strategy achieves +1dB PSNR improvement over state-of-the-art defocus deblurring methods. These high-quality DP views can be used for other DP-based applications, such as reflection removal.
  arXiv  Detail & Related papers  (2021-08-11T14:45:15Z)
• Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems [56.154190098338965]
  We show that our proposed time multiplexed coded aperture (TMCA) can be optimized end-to-end. TMCA induces better coded snapshots enabling superior reconstructions in two different applications: compressive light field imaging and hyperspectral imaging. This codification outperforms the state-of-the-art compressive imaging systems by more than 4dB in those applications.
  arXiv  Detail & Related papers  (2021-04-06T22:42:34Z)
• Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data [42.06108142009718]
  Recent work has shown impressive results on data-driven deblurring using the two-image views available on modern dual-pixel (DP) sensors. Despite many cameras having DP sensors, only a limited number provide access to the low-level DP sensor images. We propose a procedure to generate realistic DP data synthetically.
  arXiv  Detail & Related papers  (2020-12-06T13:12:43Z)
• Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration [77.1056200937214]
  We study the formation of the DP pair which links the blur and the depth information. We propose an end-to-end DDDNet (DP-based Depth and De Network) to jointly estimate the depth and restore the image.
  arXiv  Detail & Related papers  (2020-12-01T06:53:57Z)
• Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics [72.9038524082252]
  We propose a compact single-shot monocular hyperspectral-depth (HS-D) imaging method. Our method uses a diffractive optical element (DOE), the point spread function of which changes with respect to both depth and spectrum. To facilitate learning the DOE, we present a first HS-D dataset by building a benchtop HS-D imager.
  arXiv  Detail & Related papers  (2020-09-01T14:19:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.