Robust super-resolution depth imaging via a multi-feature fusion deep network

• URL: http://arxiv.org/abs/2011.11444v2
• Date: Mon, 1 Feb 2021 11:25:46 GMT
• Title: Robust super-resolution depth imaging via a multi-feature fusion deep network
• Authors: Alice Ruget, Stephen McLaughlin, Robert K. Henderson, Istvan Gyongy, Abderrahim Halimi and Jonathan Leach
• Abstract summary: Light detection and ranging (LIDAR) via single-photon sensitive detector (SPAD) arrays is an emerging technology that enables the acquisition of depth images at high frame rates. We develop a deep network built specifically to take advantage of the multiple features that can be extracted from a camera's histogram data. We apply the network to a range of 3D data, demonstrating denoising and a four-fold resolution enhancement of depth.
• Score: 2.351601888896043
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Three-dimensional imaging plays an important role in imaging applications where it is necessary to record depth. The number of applications that use depth imaging is increasing rapidly, and examples include self-driving autonomous vehicles and auto-focus assist on smartphone cameras. Light detection and ranging (LIDAR) via single-photon sensitive detector (SPAD) arrays is an emerging technology that enables the acquisition of depth images at high frame rates. However, the spatial resolution of this technology is typically low in comparison to the intensity images recorded by conventional cameras. To increase the native resolution of depth images from a SPAD camera, we develop a deep network built specifically to take advantage of the multiple features that can be extracted from a camera's histogram data. The network is designed for a SPAD camera operating in a dual-mode such that it captures alternate low resolution depth and high resolution intensity images at high frame rates, thus the system does not require any additional sensor to provide intensity images. The network then uses the intensity images and multiple features extracted from downsampled histograms to guide the upsampling of the depth. Our network provides significant image resolution enhancement and image denoising across a wide range of signal-to-noise ratios and photon levels. We apply the network to a range of 3D data, demonstrating denoising and a four-fold resolution enhancement of depth.

Related papers

• Pixel-Aligned Multi-View Generation with Depth Guided Decoder [86.1813201212539]
  We propose a novel method for pixel-level image-to-multi-view generation. Unlike prior work, we incorporate attention layers across multi-view images in the VAE decoder of a latent video diffusion model. Our model enables better pixel alignment across multi-view images.
  arXiv  Detail & Related papers  (2024-08-26T04:56:41Z)
• Depth Map Denoising Network and Lightweight Fusion Network for Enhanced 3D Face Recognition [61.27785140017464]
  We introduce an innovative Depth map denoising network (DMDNet) based on the Denoising Implicit Image Function (DIIF) to reduce noise. We further design a powerful recognition network called Lightweight Depth and Normal Fusion network (LDNFNet) to learn unique and complementary features between different modalities.
  arXiv  Detail & Related papers  (2024-01-01T10:46:42Z)
• Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography [54.36608424943729]
  We show that in a ''long-burst'', forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. We devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion.
  arXiv  Detail & Related papers  (2022-12-22T18:54:34Z)
• Video super-resolution for single-photon LIDAR [0.0]
  3D Time-of-Flight (ToF) image sensors are used widely in applications such as self-driving cars, Augmented Reality (AR) and robotics. In this paper, we use synthetic depth sequences to train a 3D Convolutional Neural Network (CNN) for denoising and upscaling (x4) depth data. With GPU acceleration, frames are processed at >30 frames per second, making the approach suitable for low-latency imaging, as required for obstacle avoidance.
  arXiv  Detail & Related papers  (2022-10-19T11:33:29Z)
• Simulating single-photon detector array sensors for depth imaging [2.497104612216142]
  Single-Photon Avalanche Detector (SPAD) arrays are a rapidly emerging technology. We establish a robust yet simple numerical procedure that establishes the fundamental limits to depth imaging with SPAD arrays.
  arXiv  Detail & Related papers  (2022-10-07T13:23:34Z)
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction [91.43066633305662]
  We propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD) Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time.
  arXiv  Detail & Related papers  (2022-03-09T17:20:18Z)
• Single image deep defocus estimation and its applications [82.93345261434943]
  We train a deep neural network to classify image patches into one of the 20 levels of blurriness. The trained model is used to determine the patch blurriness which is then refined by applying an iterative weighted guided filter. The result is a defocus map that carries the information of the degree of blurriness for each pixel.
  arXiv  Detail & Related papers  (2021-07-30T06:18:16Z)
• Self-supervised Depth Denoising Using Lower- and Higher-quality RGB-D sensors [8.34403807284064]
  We propose a self-supervised depth denoising approach to denoise and refine depth coming from a low quality sensor. We record simultaneous RGB-D sequences with unzynchronized lower- and higher-quality cameras and solve a challenging problem of aligning sequences both temporally and spatially. We then learn a deep neural network to denoise the lower-quality depth using the matched higher-quality data as a source of supervision signal.
  arXiv  Detail & Related papers  (2020-09-10T11:18:11Z)
• Quanta Burst Photography [15.722085082004934]
  Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting individual incident photons. We present quanta burst photography, a computational photography technique that leverages SPCs as passive imaging devices for photography in challenging conditions.
  arXiv  Detail & Related papers  (2020-06-21T16:20:29Z)
• Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
  We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object. We first estimate per-view depth maps using a deep multi-view stereo network. These depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture.
  arXiv  Detail & Related papers  (2020-03-27T21:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.