Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection

• URL: http://arxiv.org/abs/2402.18493v1
• Date: Wed, 28 Feb 2024 17:21:02 GMT
• Title: Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection
• Authors: Xun Huang, Hai Wu, Xin Li, Xiaoliang Fan, Chenglu Wen, Cheng Wang
• Abstract summary: Previous research has attempted to address this by simulating the noise from rain to improve the robustness of detection models. We propose a novel rain simulation method, termed DRET, that unifies Dynamics and Rainy Environment Theory. We also present a Sunny-to-Rainy Knowledge Distillation approach to enhance 3D detection under rainy conditions.
• Score: 26.278415287992964
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LiDAR-based 3D object detection models have traditionally struggled under rainy conditions due to the degraded and noisy scanning signals. Previous research has attempted to address this by simulating the noise from rain to improve the robustness of detection models. However, significant disparities exist between simulated and actual rain-impacted data points. In this work, we propose a novel rain simulation method, termed DRET, that unifies Dynamics and Rainy Environment Theory to provide a cost-effective means of expanding the available realistic rain data for 3D detection training. Furthermore, we present a Sunny-to-Rainy Knowledge Distillation (SRKD) approach to enhance 3D detection under rainy conditions. Extensive experiments on the WaymoOpenDataset large-scale dataset show that, when combined with the state-of-the-art DSVT model and other classical 3D detectors, our proposed framework demonstrates significant detection accuracy improvements, without losing efficiency. Remarkably, our framework also improves detection capabilities under sunny conditions, therefore offering a robust solution for 3D detection regardless of whether the weather is rainy or sunny

Related papers

• Towards Robust 3D Object Detection In Rainy Conditions [10.920640666237833]
  We propose a framework for improving the robustness of LiDAR-based 3D object detectors against road spray. Our approach uses a state-of-the-art adverse weather detection network to filter out spray from the LiDAR point cloud. In addition to adverse weather filtering, we explore the use of radar targets to further filter false positive detections.
  arXiv  Detail & Related papers  (2023-10-02T07:34:15Z)
• Radar Enlighten the Dark: Enhancing Low-Visibility Perception for Automated Vehicles with Camera-Radar Fusion [8.946655323517094]
  We propose a novel transformer-based 3D object detection model "REDFormer" to tackle low visibility conditions. Our model outperforms state-of-the-art (SOTA) models on classification and detection accuracy.
  arXiv  Detail & Related papers  (2023-05-27T00:47:39Z)
• Boosting 3D Object Detection by Simulating Multimodality on Point Clouds [51.87740119160152]
  This paper presents a new approach to boost a single-modality (LiDAR) 3D object detector by teaching it to simulate features and responses that follow a multi-modality (LiDAR-image) detector. The approach needs LiDAR-image data only when training the single-modality detector, and once well-trained, it only needs LiDAR data at inference. Experimental results on the nuScenes dataset show that our approach outperforms all SOTA LiDAR-only 3D detectors.
  arXiv  Detail & Related papers  (2022-06-30T01:44:30Z)
• LiDAR Snowfall Simulation for Robust 3D Object Detection [116.10039516404743]
  We propose a physically based method to simulate the effect of snowfall on real clear-weather LiDAR point clouds. Our method samples snow particles in 2D space for each LiDAR line and uses the induced geometry to modify the measurement for each LiDAR beam. We use our simulation to generate partially synthetic snowy LiDAR data and leverage these data for training 3D object detection models that are robust to snowfall.
  arXiv  Detail & Related papers  (2022-03-28T21:48:26Z)
• Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather [92.84066576636914]
  This work addresses the challenging task of LiDAR-based 3D object detection in foggy weather. We tackle this problem by simulating physically accurate fog into clear-weather scenes. We are the first to provide strong 3D object detection baselines on the Seeing Through Fog dataset.
  arXiv  Detail & Related papers  (2021-08-11T14:37:54Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)
• Semi-Supervised Video Deraining with Dynamic Rain Generator [59.71640025072209]
  This paper proposes a new semi-supervised video deraining method, in which a dynamic rain generator is employed to fit the rain layer. Specifically, such dynamic generator consists of one emission model and one transition model to simultaneously encode the spatially physical structure and temporally continuous changes of rain streaks. Various prior formats are designed for the labeled synthetic and unlabeled real data, so as to fully exploit the common knowledge underlying them.
  arXiv  Detail & Related papers  (2021-03-14T14:28:57Z)
• From Rain Generation to Rain Removal [67.71728610434698]
  We build a full Bayesian generative model for rainy image where the rain layer is parameterized as a generator. We employ the variational inference framework to approximate the expected statistical distribution of rainy image. Comprehensive experiments substantiate that the proposed model can faithfully extract the complex rain distribution.
  arXiv  Detail & Related papers  (2020-08-08T18:56:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.