WeatherProof: Leveraging Language Guidance for Semantic Segmentation in Adverse Weather

• URL: http://arxiv.org/abs/2403.14874v2
• Date: Tue, 7 May 2024 21:25:06 GMT
• Title: WeatherProof: Leveraging Language Guidance for Semantic Segmentation in Adverse Weather
• Authors: Blake Gella, Howard Zhang, Rishi Upadhyay, Tiffany Chang, Nathan Wei, Matthew Waliman, Yunhao Ba, Celso de Melo, Alex Wong, Achuta Kadambi,
• Abstract summary: We propose a method to infer semantic segmentation maps from images captured under adverse weather conditions. We begin by examining existing models on images degraded by weather conditions such as rain, fog, or snow. We propose WeatherProof, the first semantic segmentation dataset with accurate clear and adverse weather image pairs.
• Score: 8.902960772665482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a method to infer semantic segmentation maps from images captured under adverse weather conditions. We begin by examining existing models on images degraded by weather conditions such as rain, fog, or snow, and found that they exhibit a large performance drop as compared to those captured under clear weather. To control for changes in scene structures, we propose WeatherProof, the first semantic segmentation dataset with accurate clear and adverse weather image pairs that share an underlying scene. Through this dataset, we analyze the error modes in existing models and found that they were sensitive to the highly complex combination of different weather effects induced on the image during capture. To improve robustness, we propose a way to use language as guidance by identifying contributions of adverse weather conditions and injecting that as "side information". Models trained using our language guidance exhibit performance gains by up to 10.2% in mIoU on WeatherProof, up to 8.44% in mIoU on the widely used ACDC dataset compared to standard training techniques, and up to 6.21% in mIoU on the ACDC dataset as compared to previous SOTA methods.

Related papers

• SemiDDM-Weather: A Semi-supervised Learning Framework for All-in-one Adverse Weather Removal [57.52777076116241]
  Adverse weather removal aims to restore clear vision under adverse weather conditions. This paper presents a pioneering semi-supervised all-in-one adverse weather removal framework built on the teacher-student network.
  arXiv  Detail & Related papers  (2024-09-29T12:12:22Z)
• Enhancing Autonomous Vehicle Perception in Adverse Weather through Image Augmentation during Semantic Segmentation Training [0.0]
  We trained encoder-decoder UNet models to perform semantic segmentation augmentations. Models trained on weather data have significantly lower losses than those trained on augmented data in all conditions except for clear days.
  arXiv  Detail & Related papers  (2024-08-14T00:08:28Z)
• Boosting Adverse Weather Crowd Counting via Multi-queue Contrastive Learning [8.692139673789555]
  We propose a two-stage crowd counting method to enhance the model's robustness in adverse weather. In the first stage, we introduce a multi-queue MoCo contrastive learning strategy to tackle the problem of weather class imbalance. In the second stage, we propose to refine the representations under the guidance of contrastive learning, enabling the conversion of the weather-aware representations to the normal weather domain.
  arXiv  Detail & Related papers  (2024-08-12T07:13:08Z)
• ACE Metric: Advection and Convection Evaluation for Accurate Weather Forecasting [7.016835396874093]
  We propose the advection and convection Error (ACE) metric to assess how well models predict advection and convection. We have validated the ACE evaluation metric on the WeatherBench2 and MovingMNIST datasets.
  arXiv  Detail & Related papers  (2024-06-07T06:49:59Z)
• Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal [53.15046196592023]
  We introduce test-time adaptation into adverse weather removal in videos. We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
  arXiv  Detail & Related papers  (2024-03-12T14:21:30Z)
• WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather [9.619700283574533]
  We introduce a general paired-training method that leads to improved performance on images in adverse weather conditions. We create the first semantic segmentation dataset with accurate clear and adverse weather image pairs. We find that training on these paired clear and adverse weather frames which share an underlying scene results in improved performance on adverse weather data.
  arXiv  Detail & Related papers  (2023-12-15T04:57:54Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• Counting Crowds in Bad Weather [68.50690406143173]
  We propose a method for robust crowd counting in adverse weather scenarios. Our model learns effective features and adaptive queries to account for large appearance variations. Experimental results show that the proposed algorithm is effective in counting crowds under different weather types on benchmark datasets.
  arXiv  Detail & Related papers  (2023-06-02T00:00:09Z)
• Deep-Learning-Based Precipitation Nowcasting with Ground Weather Station Data and Radar Data [14.672132394870445]
  We propose ASOC, a novel attentive method for effectively exploiting ground-based meteorological observations from multiple weather stations. ASOC is designed to capture temporal dynamics of the observations and also contextual relationships between them. We show that such a combination improves the average critical success index (CSI) of predicting heavy (at least 10 mm/hr) and light (at least 1 mm/hr) rainfall events at 1-6 hr lead times by 5.7%.
  arXiv  Detail & Related papers  (2022-10-20T14:59:58Z)
• TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions [77.20136060506906]
  We propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder. TransWeather achieves significant improvements across multiple test datasets over both All-in-One network. It is validated on real world test images and found to be more effective than previous methods.
  arXiv  Detail & Related papers  (2021-11-29T18:57:09Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.