WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather

• URL: http://arxiv.org/abs/2312.09534v1
• Date: Fri, 15 Dec 2023 04:57:54 GMT
• Title: WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather
• Authors: Blake Gella, Howard Zhang, Rishi Upadhyay, Tiffany Chang, Matthew Waliman, Yunhao Ba, Alex Wong, Achuta Kadambi
• Abstract summary: We introduce a general paired-training method that leads to improved performance on images in adverse weather conditions. We create the first semantic segmentation dataset with accurate clear and adverse weather image pairs. We find that training on these paired clear and adverse weather frames which share an underlying scene results in improved performance on adverse weather data.
• Score: 9.619700283574533
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The introduction of large, foundational models to computer vision has led to drastically improved performance on the task of semantic segmentation. However, these existing methods exhibit a large performance drop when testing on images degraded by weather conditions such as rain, fog, or snow. We introduce a general paired-training method that can be applied to all current foundational model architectures that leads to improved performance on images in adverse weather conditions. To this end, we create the WeatherProof Dataset, the first semantic segmentation dataset with accurate clear and adverse weather image pairs, which not only enables our new training paradigm, but also improves the evaluation of the performance gap between clear and degraded segmentation. We find that training on these paired clear and adverse weather frames which share an underlying scene results in improved performance on adverse weather data. With this knowledge, we propose a training pipeline which accentuates the advantages of paired-data training using consistency losses and language guidance, which leads to performance improvements by up to 18.4% as compared to standard training procedures.

Related papers

• SemiDDM-Weather: A Semi-supervised Learning Framework for All-in-one Adverse Weather Removal [57.52777076116241]
  Adverse weather removal aims to restore clear vision under adverse weather conditions. This paper presents a pioneering semi-supervised all-in-one adverse weather removal framework built on the teacher-student network.
  arXiv  Detail & Related papers  (2024-09-29T12:12:22Z)
• Towards Real-World Adverse Weather Image Restoration: Enhancing Clearness and Semantics with Vision-Language Models [85.53913950137175]
  We formulate a semi-supervised learning framework employing vision-language models to enhance restoration performance in real-world conditions. For clearness enhancement, we use real-world data, utilizing a dual-step strategy with pseudo-labels assessed by vision-language models and weather prompt learning. For semantic enhancement, we integrate real-world data by adjusting weather conditions in vision-language model descriptions while preserving semantic meaning.
  arXiv  Detail & Related papers  (2024-09-03T17:56:51Z)
• Boosting Adverse Weather Crowd Counting via Multi-queue Contrastive Learning [8.692139673789555]
  We propose a two-stage crowd counting method to enhance the model's robustness in adverse weather. In the first stage, we introduce a multi-queue MoCo contrastive learning strategy to tackle the problem of weather class imbalance. In the second stage, we propose to refine the representations under the guidance of contrastive learning, enabling the conversion of the weather-aware representations to the normal weather domain.
  arXiv  Detail & Related papers  (2024-08-12T07:13:08Z)
• Rethinking Data Augmentation for Robust LiDAR Semantic Segmentation in Adverse Weather [21.040167521248772]
  Existing LiDAR semantic segmentation methods often struggle with performance declines in adverse weather conditions. Previous work has addressed this issue by simulating adverse weather or employing universal data augmentation during training. We propose new strategic data augmentation techniques to pinpoint the main causes of performance degradation. Our method achieves a notable 39.5 mIoU on the Semantic KITTI-to-SemanticSTF benchmark, improving the baseline by 8.1%p and establishing a new state-of-the-art.
  arXiv  Detail & Related papers  (2024-07-02T14:19:51Z)
• WeatherProof: Leveraging Language Guidance for Semantic Segmentation in Adverse Weather [8.902960772665482]
  We propose a method to infer semantic segmentation maps from images captured under adverse weather conditions. We begin by examining existing models on images degraded by weather conditions such as rain, fog, or snow. We propose WeatherProof, the first semantic segmentation dataset with accurate clear and adverse weather image pairs.
  arXiv  Detail & Related papers  (2024-03-21T22:46:27Z)
• Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal [53.15046196592023]
  We introduce test-time adaptation into adverse weather removal in videos. We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
  arXiv  Detail & Related papers  (2024-03-12T14:21:30Z)
• Learning Real-World Image De-Weathering with Imperfect Supervision [57.748585821252824]
  Existing real-world de-weathering datasets often exhibit inconsistent illumination, position, and textures between the ground-truth images and the input degraded images. We develop a Consistent Label Constructor (CLC) to generate a pseudo-label as consistent as possible with the input degraded image. We combine the original imperfect labels and pseudo-labels to jointly supervise the de-weathering model by the proposed Information Allocation Strategy.
  arXiv  Detail & Related papers  (2023-10-23T14:02:57Z)
• WeatherDepth: Curriculum Contrastive Learning for Self-Supervised Depth Estimation under Adverse Weather Conditions [42.99525455786019]
  We propose WeatherDepth, a self-supervised robust depth estimation model with curriculum contrastive learning. The proposed solution is proven to be easily incorporated into various architectures and demonstrates state-of-the-art (SoTA) performance on both synthetic and real weather datasets.
  arXiv  Detail & Related papers  (2023-10-09T09:26:27Z)
• Counting Crowds in Bad Weather [68.50690406143173]
  We propose a method for robust crowd counting in adverse weather scenarios. Our model learns effective features and adaptive queries to account for large appearance variations. Experimental results show that the proposed algorithm is effective in counting crowds under different weather types on benchmark datasets.
  arXiv  Detail & Related papers  (2023-06-02T00:00:09Z)
• Unsupervised Restoration of Weather-affected Images using Deep Gaussian Process-based CycleGAN [92.15895515035795]
  We describe an approach for supervising deep networks that are based on CycleGAN. We introduce new losses for training CycleGAN that lead to more effective training, resulting in high-quality reconstructions. We demonstrate that the proposed method can be effectively applied to different restoration tasks like de-raining, de-hazing and de-snowing.
  arXiv  Detail & Related papers  (2022-04-23T01:30:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.