DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detection

• URL: http://arxiv.org/abs/2505.07398v1
• Date: Mon, 12 May 2025 09:53:00 GMT
• Title: DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detection
• Authors: Mingqian Ji, Jian Yang, Shanshan Zhang,
• Abstract summary: State-of-the-art LiDAR-camera 3D object detectors usually focus on feature fusion.<n>We are the first to observe that different modalities play different roles as depth varies via statistical analysis and visualization.<n>We propose a Depth-Aware Hybrid Feature Fusion strategy that guides the weights of point cloud and RGB image modalities.
• Score: 32.07206206508925
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-of-the-art LiDAR-camera 3D object detectors usually focus on feature fusion. However, they neglect the factor of depth while designing the fusion strategy. In this work, we are the first to observe that different modalities play different roles as depth varies via statistical analysis and visualization. Based on this finding, we propose a Depth-Aware Hybrid Feature Fusion (DepthFusion) strategy that guides the weights of point cloud and RGB image modalities by introducing depth encoding at both global and local levels. Specifically, the Depth-GFusion module adaptively adjusts the weights of image Bird's-Eye-View (BEV) features in multi-modal global features via depth encoding. Furthermore, to compensate for the information lost when transferring raw features to the BEV space, we propose a Depth-LFusion module, which adaptively adjusts the weights of original voxel features and multi-view image features in multi-modal local features via depth encoding. Extensive experiments on the nuScenes and KITTI datasets demonstrate that our DepthFusion method surpasses previous state-of-the-art methods. Moreover, our DepthFusion is more robust to various kinds of corruptions, outperforming previous methods on the nuScenes-C dataset.

Related papers

• UniCT Depth: Event-Image Fusion Based Monocular Depth Estimation with Convolution-Compensated ViT Dual SA Block [6.994911870644179]
  We propose UniCT Depth, an event-image fusion method that unifies CNNs and Transformers to model local and global features.<n>We show that UniCT Depth outperforms existing image, event, and fusion-based monocular depth estimation methods across key metrics.
  arXiv  Detail & Related papers  (2025-07-26T13:29:48Z)
• Language-Depth Navigated Thermal and Visible Image Fusion [11.473316170288166]
  Existing thermal-visible image fusion mainly focuses on detection tasks, ignoring other critical information such as depth.<n>We introduce a text-guided and depth-driven infrared and visible image fusion network.<n>This supports precise recognition and efficient operations in applications such as autonomous driving and rescue missions.
  arXiv  Detail & Related papers  (2025-03-11T17:55:22Z)
• DepthLab: From Partial to Complete [80.58276388743306]
  Missing values remain a common challenge for depth data across its wide range of applications.<n>This work bridges this gap with DepthLab, a foundation depth inpainting model powered by image diffusion priors.<n>Our approach proves its worth in various downstream tasks, including 3D scene inpainting, text-to-3D scene generation, sparse-view reconstruction with DUST3R, and LiDAR depth completion.
  arXiv  Detail & Related papers  (2024-12-24T04:16:38Z)
• Depth Estimation Based on 3D Gaussian Splatting Siamese Defocus [14.354405484663285]
  We propose a self-supervised framework based on 3D Gaussian splatting and Siamese networks for depth estimation in 3D geometry.<n>The proposed framework has been validated on both artificially synthesized and real blurred datasets.
  arXiv  Detail & Related papers  (2024-09-18T21:36:37Z)
• Robust Depth Enhancement via Polarization Prompt Fusion Tuning [112.88371907047396]
  We present a framework that leverages polarization imaging to improve inaccurate depth measurements from various depth sensors. Our method first adopts a learning-based strategy where a neural network is trained to estimate a dense and complete depth map from polarization data and a sensor depth map from different sensors. To further improve the performance, we propose a Polarization Prompt Fusion Tuning (PPFT) strategy to effectively utilize RGB-based models pre-trained on large-scale datasets.
  arXiv  Detail & Related papers  (2024-04-05T17:55:33Z)
• DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection [83.18142309597984]
  Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. We develop a family of generic multi-modal 3D detection models named DeepFusion, which is more accurate than previous methods.
  arXiv  Detail & Related papers  (2022-03-15T18:46:06Z)
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction [91.43066633305662]
  We propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD) Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time.
  arXiv  Detail & Related papers  (2022-03-09T17:20:18Z)
• High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion [84.24973877109181]
  We propose a novel attention-based hierarchical multi-modal fusion network for guided DSR. We show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.
  arXiv  Detail & Related papers  (2021-04-04T03:28:33Z)
• Video Depth Estimation by Fusing Flow-to-Depth Proposals [65.24533384679657]
  We present an approach with a differentiable flow-to-depth layer for video depth estimation. The model consists of a flow-to-depth layer, a camera pose refinement module, and a depth fusion network. Our approach outperforms state-of-the-art depth estimation methods, and has reasonable cross dataset generalization capability.
  arXiv  Detail & Related papers  (2019-12-30T10:45:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.