Video super-resolution for single-photon LIDAR

• URL: http://arxiv.org/abs/2210.10474v1
• Date: Wed, 19 Oct 2022 11:33:29 GMT
• Title: Video super-resolution for single-photon LIDAR
• Authors: Germ\'an Mora Mart\'in, Stirling Scholes, Alice Ruget, Robert K. Henderson, Jonathan Leach, Istvan Gyongy
• Abstract summary: 3D Time-of-Flight (ToF) image sensors are used widely in applications such as self-driving cars, Augmented Reality (AR) and robotics. In this paper, we use synthetic depth sequences to train a 3D Convolutional Neural Network (CNN) for denoising and upscaling (x4) depth data. With GPU acceleration, frames are processed at >30 frames per second, making the approach suitable for low-latency imaging, as required for obstacle avoidance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D Time-of-Flight (ToF) image sensors are used widely in applications such as self-driving cars, Augmented Reality (AR) and robotics. When implemented with Single-Photon Avalanche Diodes (SPADs), compact, array format sensors can be made that offer accurate depth maps over long distances, without the need for mechanical scanning. However, array sizes tend to be small, leading to low lateral resolution, which combined with low Signal-to-Noise Ratio (SNR) levels under high ambient illumination, may lead to difficulties in scene interpretation. In this paper, we use synthetic depth sequences to train a 3D Convolutional Neural Network (CNN) for denoising and upscaling (x4) depth data. Experimental results, based on synthetic as well as real ToF data, are used to demonstrate the effectiveness of the scheme. With GPU acceleration, frames are processed at >30 frames per second, making the approach suitable for low-latency imaging, as required for obstacle avoidance.

Related papers

• Depth Map Denoising Network and Lightweight Fusion Network for Enhanced 3D Face Recognition [61.27785140017464]
  We introduce an innovative Depth map denoising network (DMDNet) based on the Denoising Implicit Image Function (DIIF) to reduce noise. We further design a powerful recognition network called Lightweight Depth and Normal Fusion network (LDNFNet) to learn unique and complementary features between different modalities.
  arXiv  Detail & Related papers  (2024-01-01T10:46:42Z)
• Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor [58.305341034419136]
  We present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. We propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results.
  arXiv  Detail & Related papers  (2023-08-28T07:56:13Z)
• Real-Time Radiance Fields for Single-Image Portrait View Synthesis [85.32826349697972]
  We present a one-shot method to infer and render a 3D representation from a single unposed image in real-time. Given a single RGB input, our image encoder directly predicts a canonical triplane representation of a neural radiance field for 3D-aware novel view synthesis via volume rendering. Our method is fast (24 fps) on consumer hardware, and produces higher quality results than strong GAN-inversion baselines that require test-time optimization.
  arXiv  Detail & Related papers  (2023-05-03T17:56:01Z)
• Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography [54.36608424943729]
  We show that in a ''long-burst'', forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. We devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion.
  arXiv  Detail & Related papers  (2022-12-22T18:54:34Z)
• Weakly-Supervised Optical Flow Estimation for Time-of-Flight [11.496094830445054]
  We propose a training algorithm, which allows to supervise Optical Flow networks directly on the reconstructed depth. We demonstrate that this approach enables the training of OF networks to align raw iToF measurements and compensate motion artifacts in the iToF depth images.
  arXiv  Detail & Related papers  (2022-10-11T09:47:23Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution [96.89588203312451]
  This paper presents a new parametric representation called Deep Parametric 3D Filters (DP3DF) DP3DF incorporates local information to enable simultaneous denoising, illumination enhancement, and SR efficiently in a single encoder-and-decoder network. Also, a dynamic residual frame is jointly learned with the DP3DF via a shared backbone to further boost the SR quality.
  arXiv  Detail & Related papers  (2022-07-05T03:57:25Z)
• A Low Memory Footprint Quantized Neural Network for Depth Completion of Very Sparse Time-of-Flight Depth Maps [14.885472968649937]
  We simulate ToF datasets for indoor 3D perception with challenging sparsity levels. Our model achieves optimal depth map quality by means of input pre-processing and carefully tuned training. We also achieve low memory footprint for weights and activations by means of mixed precision quantization-at-training techniques.
  arXiv  Detail & Related papers  (2022-05-25T17:11:31Z)
• Single-Photon Structured Light [31.614032717665832]
  "Single-Photon Structured Light" works by sensing binary images that indicates the presence or absence of photon arrivals during each exposure. We develop novel temporal sequences using error correction codes that are designed to be robust to short-range effects like projector and camera defocus. Our lab prototype is capable of 3D imaging in challenging scenarios involving objects with extremely low albedo or undergoing fast motion.
  arXiv  Detail & Related papers  (2022-04-11T17:57:04Z)
• High-speed object detection with a single-photon time-of-flight image sensor [2.648554238948439]
  We present results from a portable SPAD camera system that outputs 16-bin photon timing histograms with 64x32 spatial resolution. The results are relevant for safety-critical computer vision applications which would benefit from faster than human reaction times.
  arXiv  Detail & Related papers  (2021-07-28T14:53:44Z)
• Robust super-resolution depth imaging via a multi-feature fusion deep network [2.351601888896043]
  Light detection and ranging (LIDAR) via single-photon sensitive detector (SPAD) arrays is an emerging technology that enables the acquisition of depth images at high frame rates. We develop a deep network built specifically to take advantage of the multiple features that can be extracted from a camera's histogram data. We apply the network to a range of 3D data, demonstrating denoising and a four-fold resolution enhancement of depth.
  arXiv  Detail & Related papers  (2020-11-20T14:24:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.