RayD3D: Distilling Depth Knowledge Along the Ray for Robust Multi-View 3D Object Detection

• URL: http://arxiv.org/abs/2603.07493v1
• Date: Sun, 08 Mar 2026 06:27:21 GMT
• Title: RayD3D: Distilling Depth Knowledge Along the Ray for Robust Multi-View 3D Object Detection
• Abstract summary: Multi-view 3D detection with bird's eye view (BEV) is crucial for autonomous driving and robotics.<n>We propose RayD3D, which transfers crucial depth knowledge along the ray.<n>Ray-based Contrastive Distillation (RCD) incorporates contrastive learning into distillation by sampling along the ray to learn how LiDAR accurately locates objects.<n>Ray-based Weighted Distillation (RWD) adaptively adjusts distillation weight based on the ray to minimize the interference of depth-irrelevant information in LiDAR.
• Score: 26.899410431099042
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-view 3D detection with bird's eye view (BEV) is crucial for autonomous driving and robotics, but its robustness in real-world is limited as it struggles to predict accurate depth values. A mainstream solution, cross-modal distillation, transfers depth information from LiDAR to camera models but also unintentionally transfers depth-irrelevant information (e.g. LiDAR density). To mitigate this issue, we propose RayD3D, which transfers crucial depth knowledge along the ray: a line projecting from the camera to true location of an object. It is based on the fundamental imaging principle that predicted location of this object can only vary along this ray, which is finally determined by predicted depth value. Therefore, distilling along the ray enables more effective depth information transfer. More specifically, we design two ray-based distillation modules. Ray-based Contrastive Distillation (RCD) incorporates contrastive learning into distillation by sampling along the ray to learn how LiDAR accurately locates objects. Ray-based Weighted Distillation (RWD) adaptively adjusts distillation weight based on the ray to minimize the interference of depth-irrelevant information in LiDAR. For validation, we widely apply RayD3D into three representative types of BEV-based models, including BEVDet, BEVDepth4D, and BEVFormer. Our method is trained on clean NuScenes, and tested on both clean NuScenes and RoboBEV with a variety types of data corruptions. Our method significantly improves the robustness of all the three base models in all scenarios without increasing inference costs, and achieves the best when compared to recently released multi-view and distillation models.

Related papers

• Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection [41.41435217865119]
  Cross-modality knowledge distillation could effectively transfer depth information from LiDAR to image-based network.<n>However, modality gap between image and LiDAR seriously limits its accuracy.<n>We propose a selective learning approach named MonoSTL to overcome these issues.
  arXiv  Detail & Related papers  (2026-03-08T05:05:07Z)
• L3DR: 3D-aware LiDAR Diffusion and Rectification [85.5914944339043]
  Range-view (RV) based LiDAR diffusion has recently made huge strides towards 2D photo-realism.<n>However, it neglects 3D geometry realism and often generates various RV artifacts such as depth bleeding and wavy surfaces.<n>We design L3DR, a 3D-aware LiDAR Diffusion and Rectification framework that can regress and cancel RV artifacts in 3D space.
  arXiv  Detail & Related papers  (2026-02-22T06:31:58Z)
• LiRCDepth: Lightweight Radar-Camera Depth Estimation via Knowledge Distillation and Uncertainty Guidance [5.9796425689255255]
  LiRCDepth is a lightweight radar-camera depth estimation model.<n>We incorporate knowledge distillation to enhance the training process.<n>The model achieves a 6.6% improvement in MAE on the nuScenes dataset.
  arXiv  Detail & Related papers  (2024-12-20T22:34:47Z)
• VFMM3D: Releasing the Potential of Image by Vision Foundation Model for Monocular 3D Object Detection [80.62052650370416]
  monocular 3D object detection holds significant importance across various applications, including autonomous driving and robotics. In this paper, we present VFMM3D, an innovative framework that leverages the capabilities of Vision Foundation Models (VFMs) to accurately transform single-view images into LiDAR point cloud representations.
  arXiv  Detail & Related papers  (2024-04-15T03:12:12Z)
• Sparse Points to Dense Clouds: Enhancing 3D Detection with Limited LiDAR Data [68.18735997052265]
  We propose a balanced approach that combines the advantages of monocular and point cloud-based 3D detection. Our method requires only a small number of 3D points, that can be obtained from a low-cost, low-resolution sensor. The accuracy of 3D detection improves by 20% compared to the state-of-the-art monocular detection methods.
  arXiv  Detail & Related papers  (2024-04-10T03:54:53Z)
• Ray Denoising: Depth-aware Hard Negative Sampling for Multi-view 3D Object Detection [46.041193889845474]
  Ray Denoising is an innovative method that enhances detection accuracy by strategically sampling along camera rays to construct hard negative examples. Ray Denoising is designed as a plug-and-play module, compatible with any DETR-style multi-view 3D detectors. It achieves a 1.9% improvement in mean Average Precision (mAP) over the state-of-the-art StreamPETR method on the NuScenes dataset.
  arXiv  Detail & Related papers  (2024-02-06T02:17:44Z)
• RayMVSNet++: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo [21.209964556493368]
  RayMVSNet learns sequential prediction of a 1D implicit field along each camera ray with the zero-crossing point indicating scene depth. RayMVSNet++ achieves state-of-the-art performance on the ScanNet dataset.
  arXiv  Detail & Related papers  (2023-07-16T02:10:47Z)
• Boosting 3D Object Detection by Simulating Multimodality on Point Clouds [51.87740119160152]
  This paper presents a new approach to boost a single-modality (LiDAR) 3D object detector by teaching it to simulate features and responses that follow a multi-modality (LiDAR-image) detector. The approach needs LiDAR-image data only when training the single-modality detector, and once well-trained, it only needs LiDAR data at inference. Experimental results on the nuScenes dataset show that our approach outperforms all SOTA LiDAR-only 3D detectors.
  arXiv  Detail & Related papers  (2022-06-30T01:44:30Z)
• LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection [96.63947479020631]
  In many real-world applications, the LiDAR points used by mass-produced robots and vehicles usually have fewer beams than that in large-scale public datasets. We propose the LiDAR Distillation to bridge the domain gap induced by different LiDAR beams for 3D object detection.
  arXiv  Detail & Related papers  (2022-03-28T17:59:02Z)
• MonoDistill: Learning Spatial Features for Monocular 3D Object Detection [80.74622486604886]
  We propose a simple and effective scheme to introduce the spatial information from LiDAR signals to the monocular 3D detectors. We use the resulting data to train a 3D detector with the same architecture as the baseline model. Experimental results show that the proposed method can significantly boost the performance of the baseline model.
  arXiv  Detail & Related papers  (2022-01-26T09:21:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.