Simulating Dual-Pixel Images From Ray Tracing For Depth Estimation

• URL: http://arxiv.org/abs/2503.11213v1
• Date: Fri, 14 Mar 2025 09:03:25 GMT
• Title: Simulating Dual-Pixel Images From Ray Tracing For Depth Estimation
• Authors: Fengchen He, Dayang Zhao, Hao Xu, Tingwei Quan, Shaoqun Zeng,
• Abstract summary: We investigate the domain gap between simulated and real DP data, and propose solutions using the Simulating DP images from ray tracing scheme.<n>The Sdirt scheme generates realistic DP images via ray tracing and integrates them into the depth estimation training pipeline.<n> Experimental results show that models trained with Sdirt-simulated images generalize better to real DP data.
• Score: 5.605804656420194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many studies utilize dual-pixel (DP) sensor phase characteristics for various applications, such as depth estimation and deblurring. However, since the DP image features are entirely determined by the camera hardware, DP-depth paired datasets are very scarce, especially when performing depth estimation on customized cameras. To overcome this, studies simulate DP images using ideal optical system models. However, these simulations often violate real optical propagation laws,leading to poor generalization to real DP data. To address this, we investigate the domain gap between simulated and real DP data, and propose solutions using the Simulating DP images from ray tracing (Sdirt) scheme. The Sdirt generates realistic DP images via ray tracing and integrates them into the depth estimation training pipeline. Experimental results show that models trained with Sdirt-simulated images generalize better to real DP data.

Related papers

• GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views [67.34073368933814]
  We propose a generalizable Gaussian Splatting approach for high-resolution image rendering under a sparse-view camera setting. We train our Gaussian parameter regression module on human-only data or human-scene data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. Experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed.
  arXiv  Detail & Related papers  (2024-11-18T08:18:44Z)
• bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction [57.199618102578576]
  We propose bit2bit, a new method for reconstructing high-quality image stacks at original resolution from sparse binary quantatemporal image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data. We present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions.
  arXiv  Detail & Related papers  (2024-10-30T17:30:35Z)
• Let's Roll: Synthetic Dataset Analysis for Pedestrian Detection Across Different Shutter Types [7.0441427250832644]
  This paper studies the impact of different shutter mechanisms on machine learning (ML) object detection models on a synthetic dataset. In particular, we train and evaluate mainstream detection models with our synthetically-generated paired GS and RS datasets.
  arXiv  Detail & Related papers  (2023-09-15T04:07:42Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• DeepRM: Deep Recurrent Matching for 6D Pose Refinement [77.34726150561087]
  DeepRM is a novel recurrent network architecture for 6D pose refinement. The architecture incorporates LSTM units to propagate information through each refinement step. DeepRM achieves state-of-the-art performance on two widely accepted challenging datasets.
  arXiv  Detail & Related papers  (2022-05-28T16:18:08Z)
• Facial Depth and Normal Estimation using Single Dual-Pixel Camera [81.02680586859105]
  We introduce a DP-oriented Depth/Normal network that reconstructs the 3D facial geometry. It contains the corresponding ground-truth 3D models including depth map and surface normal in metric scale. It achieves state-of-the-art performances over recent DP-based depth/normal estimation methods.
  arXiv  Detail & Related papers  (2021-11-25T05:59:27Z)
• Physics-based Differentiable Depth Sensor Simulation [5.134435281973137]
  We introduce a novel end-to-end differentiable simulation pipeline for the generation of realistic 2.5D scans. Each module can be differentiated w.r.t sensor and scene parameters. Our simulation greatly improves the performance of the resulting models on real scans.
  arXiv  Detail & Related papers  (2021-03-30T17:59:43Z)
• Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data [42.06108142009718]
  Recent work has shown impressive results on data-driven deblurring using the two-image views available on modern dual-pixel (DP) sensors. Despite many cameras having DP sensors, only a limited number provide access to the low-level DP sensor images. We propose a procedure to generate realistic DP data synthetically.
  arXiv  Detail & Related papers  (2020-12-06T13:12:43Z)
• Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration [77.1056200937214]
  We study the formation of the DP pair which links the blur and the depth information. We propose an end-to-end DDDNet (DP-based Depth and De Network) to jointly estimate the depth and restore the image.
  arXiv  Detail & Related papers  (2020-12-01T06:53:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.