Self-Supervised Enhancement for Depth from a Lightweight ToF Sensor with Monocular Images

• URL: http://arxiv.org/abs/2506.13444v2
• Date: Tue, 17 Jun 2025 06:43:53 GMT
• Title: Self-Supervised Enhancement for Depth from a Lightweight ToF Sensor with Monocular Images
• Authors: Laiyan Ding, Hualie Jiang, Jiwei Chen, Rui Huang,
• Abstract summary: We propose a self-supervised learning framework, SelfToF, which generates detailed and scale-aware depth maps.<n>Our proposed method is both efficient and effective, as verified by extensive experiments on the NYU and ScanNet datasets.
• Score: 7.317782580649895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Depth map enhancement using paired high-resolution RGB images offers a cost-effective solution for improving low-resolution depth data from lightweight ToF sensors. Nevertheless, naively adopting a depth estimation pipeline to fuse the two modalities requires groundtruth depth maps for supervision. To address this, we propose a self-supervised learning framework, SelfToF, which generates detailed and scale-aware depth maps. Starting from an image-based self-supervised depth estimation pipeline, we add low-resolution depth as inputs, design a new depth consistency loss, propose a scale-recovery module, and finally obtain a large performance boost. Furthermore, since the ToF signal sparsity varies in real-world applications, we upgrade SelfToF to SelfToF* with submanifold convolution and guided feature fusion. Consequently, SelfToF* maintain robust performance across varying sparsity levels in ToF data. Overall, our proposed method is both efficient and effective, as verified by extensive experiments on the NYU and ScanNet datasets. The code is available at \href{https://github.com/denyingmxd/selftof}{https://github.com/denyingmxd/selftof}.

Related papers

• Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion [51.69876947593144]
  Existing methods for depth completion operate in tightly constrained settings.<n>Inspired by advances in monocular depth estimation, we reframe depth completion as an image-conditional depth map generation.<n>Marigold-DC builds on a pretrained latent diffusion model for monocular depth estimation and injects the depth observations as test-time guidance.
  arXiv  Detail & Related papers  (2024-12-18T00:06:41Z)
• Towards Single-Lens Controllable Depth-of-Field Imaging via Depth-Aware Point Spread Functions [19.312034704019634]
  Controllable Depth-of-Field (DoF) imaging commonly produces amazing visual effects based on heavy and expensive high-end lenses.<n>This work centers around two major limitations of Minimalist Optical Systems (MOS), for achieving single-lens controllable DoF imaging via computational methods.<n>A Depth-aware Controllable DoF Imaging (DCDI) framework is proposed equipped with All-in-Focus (AiF) aberration correction and monocular depth estimation.<n>With the predicted depth map, recovered image, and depth-aware PSF map inferred by Omni-Lens-Field, single-lens controllable DoF imaging is
  arXiv  Detail & Related papers  (2024-09-15T14:52:16Z)
• Robust Depth Enhancement via Polarization Prompt Fusion Tuning [112.88371907047396]
  We present a framework that leverages polarization imaging to improve inaccurate depth measurements from various depth sensors. Our method first adopts a learning-based strategy where a neural network is trained to estimate a dense and complete depth map from polarization data and a sensor depth map from different sensors. To further improve the performance, we propose a Polarization Prompt Fusion Tuning (PPFT) strategy to effectively utilize RGB-based models pre-trained on large-scale datasets.
  arXiv  Detail & Related papers  (2024-04-05T17:55:33Z)
• RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion [31.70022495622075]
  We explore a repetitive design in our image guided network to gradually and sufficiently recover depth values. In the former branch, we design a dense repetitive hourglass network (DRHN) to extract discriminative image features of complex environments. In the latter branch, we present a repetitive guidance (RG) module based on dynamic convolution, in which an efficient convolution factorization is proposed to reduce the complexity. In addition, we propose a region-aware spatial propagation network (RASPN) for further depth refinement based on the semantic prior constraint.
  arXiv  Detail & Related papers  (2023-09-01T09:11:20Z)
• RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion [28.634851863097953]
  We propose a novel two-branch end-to-end fusion network named RDFC-GAN. It takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map. The first branch employs an encoder-decoder structure, by adhering to the Manhattan world assumption. The other branch applies an RGB-depth fusion CycleGAN, adept at translating RGB imagery into detailed, textured depth maps.
  arXiv  Detail & Related papers  (2023-06-06T11:03:05Z)
• DELTAR: Depth Estimation from a Light-weight ToF Sensor and RGB Image [39.389538555506256]
  We propose DELTAR, a novel method to empower light-weight ToF sensors with the capability of measuring high resolution and accurate depth. As the core of DELTAR, a feature extractor customized for depth distribution and an attention-based neural architecture is proposed to fuse the information from the color and ToF domain efficiently. Experiments show that our method produces more accurate depth than existing frameworks designed for depth completion and depth super-resolution and achieves on par performance with a commodity-level RGB-D sensor.
  arXiv  Detail & Related papers  (2022-09-27T13:11:37Z)
• Learning an Efficient Multimodal Depth Completion Model [11.740546882538142]
  RGB image-guided sparse depth completion has attracted extensive attention recently, but still faces some problems. The proposed method can outperform some state-of-the-art methods with a lightweight architecture. The method also wins the championship in the MIPI2022 RGB+TOF depth completion challenge.
  arXiv  Detail & Related papers  (2022-08-23T07:03:14Z)
• End-to-end Learning for Joint Depth and Image Reconstruction from Diffracted Rotation [10.896567381206715]
  We propose a novel end-to-end learning approach for depth from diffracted rotation. Our approach requires a significantly less complex model and less training data, yet it is superior to existing methods in the task of monocular depth estimation.
  arXiv  Detail & Related papers  (2022-04-14T16:14:37Z)
• RGB-Depth Fusion GAN for Indoor Depth Completion [29.938869342958125]
  In this paper, we design a novel two-branch end-to-end fusion network, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map. In one branch, we propose an RGB-depth fusion GAN to transfer the RGB image to the fine-grained textured depth map. In the other branch, we adopt adaptive fusion modules named W-AdaIN to propagate the features across the two branches.
  arXiv  Detail & Related papers  (2022-03-21T10:26:38Z)
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction [91.43066633305662]
  We propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD) Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time.
  arXiv  Detail & Related papers  (2022-03-09T17:20:18Z)
• High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion [84.24973877109181]
  We propose a novel attention-based hierarchical multi-modal fusion network for guided DSR. We show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.
  arXiv  Detail & Related papers  (2021-04-04T03:28:33Z)
• Progressive Depth Learning for Single Image Dehazing [56.71963910162241]
  Existing dehazing methods often ignore the depth cues and fail in distant areas where heavier haze disturbs the visibility. We propose a deep end-to-end model that iteratively estimates image depths and transmission maps. Our approach benefits from explicitly modeling the inner relationship of image depth and transmission map, which is especially effective for distant hazy areas.
  arXiv  Detail & Related papers  (2021-02-21T05:24:18Z)
• A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection [89.88222217065858]
  We design a single stream network to use the depth map to guide early fusion and middle fusion between RGB and depth. This model is 55.5% lighter than the current lightest model and runs at a real-time speed of 32 FPS when processing a $384 times 384$ image.
  arXiv  Detail & Related papers  (2020-07-14T04:40:14Z)
• Video Depth Estimation by Fusing Flow-to-Depth Proposals [65.24533384679657]
  We present an approach with a differentiable flow-to-depth layer for video depth estimation. The model consists of a flow-to-depth layer, a camera pose refinement module, and a depth fusion network. Our approach outperforms state-of-the-art depth estimation methods, and has reasonable cross dataset generalization capability.
  arXiv  Detail & Related papers  (2019-12-30T10:45:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.