Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor

• URL: http://arxiv.org/abs/2308.14383v1
• Date: Mon, 28 Aug 2023 07:56:13 GMT
• Title: Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor
• Authors: Xinyang Liu, Yijin Li, Yanbin Teng, Hujun Bao, Guofeng Zhang, Yinda Zhang, Zhaopeng Cui
• Abstract summary: We present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. We propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results.
• Score: 58.305341034419136
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Light-weight time-of-flight (ToF) depth sensors are compact and cost-efficient, and thus widely used on mobile devices for tasks such as autofocus and obstacle detection. However, due to the sparse and noisy depth measurements, these sensors have rarely been considered for dense geometry reconstruction. In this work, we present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. Specifically, we propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor which drives the optimization by comparing with the raw sensor inputs. Moreover, in order to guarantee successful pose tracking and reconstruction, we exploit a predicted depth as an intermediate supervision and develop a coarse-to-fine optimization strategy for efficient learning of the implicit representation. At last, the temporal information is explicitly exploited to deal with the noisy signals from light-weight ToF sensors to improve the accuracy and robustness of the system. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results both on camera tracking and dense scene reconstruction. Project page: \url{https://zju3dv.github.io/tof_slam/}.

Related papers

• Characterization of point-source transient events with a rolling-shutter compressed sensing system [0.0]
  Point-source transient events (PSTEs) pose several challenges to an imaging system. Traditional imaging systems that meet these requirements are costly in terms of price, size, weight, power consumption, and data bandwidth. We develop a novel compressed sensing algorithm adapted to the rolling shutter readout of an imaging system.
  arXiv  Detail & Related papers  (2024-08-29T19:22:37Z)
• Robust Depth Enhancement via Polarization Prompt Fusion Tuning [112.88371907047396]
  We present a framework that leverages polarization imaging to improve inaccurate depth measurements from various depth sensors. Our method first adopts a learning-based strategy where a neural network is trained to estimate a dense and complete depth map from polarization data and a sensor depth map from different sensors. To further improve the performance, we propose a Polarization Prompt Fusion Tuning (PPFT) strategy to effectively utilize RGB-based models pre-trained on large-scale datasets.
  arXiv  Detail & Related papers  (2024-04-05T17:55:33Z)
• Fisheye Camera and Ultrasonic Sensor Fusion For Near-Field Obstacle Perception in Bird's-Eye-View [4.536942273206611]
  We present the first end-to-end multimodal fusion model tailored for efficient obstacle perception in a bird's-eye-view (BEV) perspective. Fisheye cameras are frequently employed for comprehensive surround-view perception, including rear-view obstacle localization. However, the performance of such cameras can significantly deteriorate in low-light conditions, during nighttime, or when subjected to intense sun glare.
  arXiv  Detail & Related papers  (2024-02-01T14:52:16Z)
• Weakly-Supervised Optical Flow Estimation for Time-of-Flight [11.496094830445054]
  We propose a training algorithm, which allows to supervise Optical Flow networks directly on the reconstructed depth. We demonstrate that this approach enables the training of OF networks to align raw iToF measurements and compensate motion artifacts in the iToF depth images.
  arXiv  Detail & Related papers  (2022-10-11T09:47:23Z)
• FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras [37.812681878193914]
  smartphones now have multimodal camera systems with time-of-flight (ToF) depth sensors and multiple color cameras. producing accurate high-resolution depth is still challenging due to the low resolution and limited active illumination power of ToF sensors. We propose an automatic calibration technique based on dense 2D/3D matching that can estimate camera parameters from a single snapshot.
  arXiv  Detail & Related papers  (2022-10-06T09:57:09Z)
• DELTAR: Depth Estimation from a Light-weight ToF Sensor and RGB Image [39.389538555506256]
  We propose DELTAR, a novel method to empower light-weight ToF sensors with the capability of measuring high resolution and accurate depth. As the core of DELTAR, a feature extractor customized for depth distribution and an attention-based neural architecture is proposed to fuse the information from the color and ToF domain efficiently. Experiments show that our method produces more accurate depth than existing frameworks designed for depth completion and depth super-resolution and achieves on par performance with a commodity-level RGB-D sensor.
  arXiv  Detail & Related papers  (2022-09-27T13:11:37Z)
• Learning Online Multi-Sensor Depth Fusion [100.84519175539378]
  SenFuNet is a depth fusion approach that learns sensor-specific noise and outlier statistics. We conduct experiments with various sensor combinations on the real-world CoRBS and Scene3D datasets.
  arXiv  Detail & Related papers  (2022-04-07T10:45:32Z)
• Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments [56.306567220448684]
  We propose a new learning based end-to-end depth prediction network which takes noisy raw I-ToF signals as well as an RGB image. We show more than 40% RMSE improvement on the final depth map compared to the baseline approach.
  arXiv  Detail & Related papers  (2021-12-07T15:04:14Z)
• ESL: Event-based Structured Light [62.77144631509817]
  Event cameras are bio-inspired sensors providing significant advantages over standard cameras. We propose a novel structured-light system using an event camera to tackle the problem of accurate and high-speed depth sensing.
  arXiv  Detail & Related papers  (2021-11-30T15:47:39Z)
• Event Guided Depth Sensing [50.997474285910734]
  We present an efficient bio-inspired event-camera-driven depth estimation algorithm. In our approach, we illuminate areas of interest densely, depending on the scene activity detected by the event camera. We show the feasibility of our approach in a simulated autonomous driving sequences and real indoor environments.
  arXiv  Detail & Related papers  (2021-10-20T11:41:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.