Cross-spectral Gated-RGB Stereo Depth Estimation

• URL: http://arxiv.org/abs/2405.12759v1
• Date: Tue, 21 May 2024 13:10:43 GMT
• Title: Cross-spectral Gated-RGB Stereo Depth Estimation
• Authors: Samuel Brucker, Stefanie Walz, Mario Bijelic, Felix Heide,
• Abstract summary: Gated cameras flood-illuminate a scene and capture the time-gated impulse response of a scene. We propose a novel stereo-depth estimation method that is capable of exploiting these multi-modal multi-view depth cues. The proposed method achieves accurate depth at long ranges, outperforming the next best existing method by 39% for ranges of 100 to 220m in MAE on accumulated LiDAR ground-truth.
• Score: 34.31592077757453
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gated cameras flood-illuminate a scene and capture the time-gated impulse response of a scene. By employing nanosecond-scale gates, existing sensors are capable of capturing mega-pixel gated images, delivering dense depth improving on today's LiDAR sensors in spatial resolution and depth precision. Although gated depth estimation methods deliver a million of depth estimates per frame, their resolution is still an order below existing RGB imaging methods. In this work, we combine high-resolution stereo HDR RCCB cameras with gated imaging, allowing us to exploit depth cues from active gating, multi-view RGB and multi-view NIR sensing -- multi-view and gated cues across the entire spectrum. The resulting capture system consists only of low-cost CMOS sensors and flood-illumination. We propose a novel stereo-depth estimation method that is capable of exploiting these multi-modal multi-view depth cues, including the active illumination that is measured by the RCCB camera when removing the IR-cut filter. The proposed method achieves accurate depth at long ranges, outperforming the next best existing method by 39% for ranges of 100 to 220m in MAE on accumulated LiDAR ground-truth. Our code, models and datasets are available at https://light.princeton.edu/gatedrccbstereo/ .

Related papers

• RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods [30.34690112905212]
  Integrating an RGB camera into a ToF imaging system has become a significant technique for perceiving the real world. This paper comprehensively reviews the works related to RGB guided ToF imaging, including network structures, learning strategies, evaluation metrics, benchmark datasets, and objective functions.
  arXiv  Detail & Related papers  (2024-05-16T17:59:58Z)
• Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution [52.582632746409665]
  We propose a novel Symmetric Uncertainty-aware Feature Transmission (SUFT) for color-guided DSR. Our method achieves superior performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-06-01T06:35:59Z)
• Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography [54.36608424943729]
  We show that in a ''long-burst'', forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. We devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion.
  arXiv  Detail & Related papers  (2022-12-22T18:54:34Z)
• Non-learning Stereo-aided Depth Completion under Mis-projection via Selective Stereo Matching [0.5067618621449753]
  We propose a non-learning depth completion method for a sparse depth map captured using a light detection and ranging (LiDAR) sensor guided by a pair of stereo images. The proposed method reduced the mean absolute error (MAE) of the depth estimation to 0.65 times and demonstrated approximately twice more accurate estimation in the long range.
  arXiv  Detail & Related papers  (2022-10-04T07:46:56Z)
• DELTAR: Depth Estimation from a Light-weight ToF Sensor and RGB Image [39.389538555506256]
  We propose DELTAR, a novel method to empower light-weight ToF sensors with the capability of measuring high resolution and accurate depth. As the core of DELTAR, a feature extractor customized for depth distribution and an attention-based neural architecture is proposed to fuse the information from the color and ToF domain efficiently. Experiments show that our method produces more accurate depth than existing frameworks designed for depth completion and depth super-resolution and achieves on par performance with a commodity-level RGB-D sensor.
  arXiv  Detail & Related papers  (2022-09-27T13:11:37Z)
• Unsupervised Visible-light Images Guided Cross-Spectrum Depth Estimation from Dual-Modality Cameras [33.77748026254935]
  Cross-spectrum depth estimation aims to provide a depth map in all illumination conditions with a pair of dual-spectrum images. In this paper, we propose an unsupervised visible-light image guided cross-spectrum (i.e., thermal and visible-light, TIR-VIS in short) depth estimation framework. Our method achieves better performance than the compared existing methods.
  arXiv  Detail & Related papers  (2022-04-30T12:58:35Z)
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction [91.43066633305662]
  We propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD) Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time.
  arXiv  Detail & Related papers  (2022-03-09T17:20:18Z)
• Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments [56.306567220448684]
  We propose a new learning based end-to-end depth prediction network which takes noisy raw I-ToF signals as well as an RGB image. We show more than 40% RMSE improvement on the final depth map compared to the baseline approach.
  arXiv  Detail & Related papers  (2021-12-07T15:04:14Z)
• Gated2Gated: Self-Supervised Depth Estimation from Gated Images [22.415893281441928]
  Gated cameras hold promise as an alternative to scanning LiDAR sensors with high-resolution 3D depth. We propose an entirely self-supervised depth estimation method that uses gated intensity profiles and temporal consistency as a training signal.
  arXiv  Detail & Related papers  (2021-12-04T19:47:38Z)
• Towards Fast and Accurate Real-World Depth Super-Resolution: Benchmark Dataset and Baseline [48.69396457721544]
  We build a large-scale dataset named "RGB-D-D" to promote the study of depth map super-resolution (SR) We provide a fast depth map super-resolution (FDSR) baseline, in which the high-frequency component adaptively decomposed from RGB image to guide the depth map SR. For the real-world LR depth maps, our algorithm can produce more accurate HR depth maps with clearer boundaries and to some extent correct the depth value errors.
  arXiv  Detail & Related papers  (2021-04-13T13:27:26Z)
• Depth Sensing Beyond LiDAR Range [84.19507822574568]
  We propose a novel three-camera system that utilizes small field of view cameras. Our system, along with our novel algorithm for computing metric depth, does not require full pre-calibration. It can output dense depth maps with practically acceptable accuracy for scenes and objects at long distances.
  arXiv  Detail & Related papers  (2020-04-07T00:09:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.