MonoWAD: Weather-Adaptive Diffusion Model for Robust Monocular 3D Object Detection

• URL: http://arxiv.org/abs/2407.16448v1
• Date: Tue, 23 Jul 2024 12:58:49 GMT
• Title: MonoWAD: Weather-Adaptive Diffusion Model for Robust Monocular 3D Object Detection
• Authors: Youngmin Oh, Hyung-Il Kim, Seong Tae Kim, Jung Uk Kim,
• Abstract summary: Existing methods mainly focus on performing 3D detection in ideal weather conditions, characterized by scenarios with clear and optimal visibility. We introduce MonoWAD, a novel weather-robust monocular 3D object detector with a weather-adaptive diffusion model. Experiments under various weather conditions demonstrate that MonoWAD achieves weather-robust monocular 3D object detection.
• Score: 22.277210748714378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monocular 3D object detection is an important challenging task in autonomous driving. Existing methods mainly focus on performing 3D detection in ideal weather conditions, characterized by scenarios with clear and optimal visibility. However, the challenge of autonomous driving requires the ability to handle changes in weather conditions, such as foggy weather, not just clear weather. We introduce MonoWAD, a novel weather-robust monocular 3D object detector with a weather-adaptive diffusion model. It contains two components: (1) the weather codebook to memorize the knowledge of the clear weather and generate a weather-reference feature for any input, and (2) the weather-adaptive diffusion model to enhance the feature representation of the input feature by incorporating a weather-reference feature. This serves an attention role in indicating how much improvement is needed for the input feature according to the weather conditions. To achieve this goal, we introduce a weather-adaptive enhancement loss to enhance the feature representation under both clear and foggy weather conditions. Extensive experiments under various weather conditions demonstrate that MonoWAD achieves weather-robust monocular 3D object detection. The code and dataset are released at https://github.com/VisualAIKHU/MonoWAD.

Related papers

• WeatherEdit: Controllable Weather Editing with 4D Gaussian Field [5.240297013713328]
  We present WeatherEdit, a novel weather editing pipeline for generating realistic weather effects in 3D scenes.<n>Our approach is structured into two key components: weather background editing and weather particle construction.<n>Experiments on multiple driving datasets demonstrate that WeatherEdit can generate diverse weather effects with controllable condition severity.
  arXiv  Detail & Related papers  (2025-05-26T19:10:47Z)
• Controllable Weather Synthesis and Removal with Video Diffusion Models [61.56193902622901]
  WeatherWeaver is a video diffusion model that synthesizes diverse weather effects directly into any input video. Our model provides precise control over weather effect intensity and supports blending various weather types, ensuring both realism and adaptability.
  arXiv  Detail & Related papers  (2025-05-01T17:59:57Z)
• WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning [69.82211470647349]
  We introduce the first generalist weather foundation model (WeatherGFM) It addresses a wide spectrum of weather understanding tasks in a unified manner. Our model can effectively handle up to ten weather understanding tasks, including weather forecasting, super-resolution, weather image translation, and post-processing.
  arXiv  Detail & Related papers  (2024-11-08T09:14:19Z)
• Multiple weather images restoration using the task transformer and adaptive mixup strategy [14.986500375481546]
  We introduce a novel multi-task severe weather removal model that can effectively handle complex weather conditions in an adaptive manner. Our model incorporates a weather task sequence generator, enabling the self-attention mechanism to selectively focus on features specific to different weather types. Our proposed model has achieved state-of-the-art performance on the publicly available dataset.
  arXiv  Detail & Related papers  (2024-09-05T04:55:40Z)
• V2X-DGW: Domain Generalization for Multi-agent Perception under Adverse Weather Conditions [36.33595322964018]
  We propose a Domain Generalization based approach, named V2X-DGW, for LiDAR-based 3D object detection on multi-agent perception system under adverse weather conditions. Our research aims to not only maintain favorable multi-agent performance in the clean weather but also promote the performance in the unseen adverse weather conditions by learning only on the clean weather data.
  arXiv  Detail & Related papers  (2024-03-17T23:29:41Z)
• Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal [53.15046196592023]
  We introduce test-time adaptation into adverse weather removal in videos. We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
  arXiv  Detail & Related papers  (2024-03-12T14:21:30Z)
• Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection [26.278415287992964]
  Previous research has attempted to address this by simulating the noise from rain to improve the robustness of detection models. We propose a novel rain simulation method, termed DRET, that unifies Dynamics and Rainy Environment Theory. We also present a Sunny-to-Rainy Knowledge Distillation approach to enhance 3D detection under rainy conditions.
  arXiv  Detail & Related papers  (2024-02-28T17:21:02Z)
• DA-RAW: Domain Adaptive Object Detection for Real-World Adverse Weather Conditions [2.048226951354646]
  We present an unsupervised domain adaptation framework for object detection in adverse weather conditions. Our method resolves the style gap by concentrating on style-related information of high-level features. Using self-supervised contrastive learning, our framework then reduces the weather gap and acquires instance features that are robust to weather corruption.
  arXiv  Detail & Related papers  (2023-09-15T04:37:28Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• MonoTDP: Twin Depth Perception for Monocular 3D Object Detection in Adverse Scenes [49.21187418886508]
  This paper proposes a monocular 3D detection model designed to perceive twin depth in adverse scenes, termed MonoTDP. We first introduce an adaptive learning strategy to aid the model in handling uncontrollable weather conditions, significantly resisting degradation caused by various degrading factors. Then, to address the depth/content loss in adverse regions, we propose a novel twin depth perception module that simultaneously estimates scene and object depth.
  arXiv  Detail & Related papers  (2023-05-18T13:42:02Z)
• Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather [92.84066576636914]
  This work addresses the challenging task of LiDAR-based 3D object detection in foggy weather. We tackle this problem by simulating physically accurate fog into clear-weather scenes. We are the first to provide strong 3D object detection baselines on the Seeing Through Fog dataset.
  arXiv  Detail & Related papers  (2021-08-11T14:37:54Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.