WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal

• URL: http://arxiv.org/abs/2303.13739v2
• Date: Thu, 4 Apr 2024 02:36:44 GMT
• Title: WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal
• Authors: Yulin Luo, Rui Zhao, Xiaobao Wei, Jinwei Chen, Yijie Lu, Shenghao Xie, Tianyu Wang, Ruiqin Xiong, Ming Lu, Shanghang Zhang,
• Abstract summary: Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. We propose Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset.
• Score: 38.257012295118145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adverse weather removal tasks like deraining, desnowing, and dehazing are usually treated as separate tasks. However, in practical autonomous driving scenarios, the type, intensity,and mixing degree of weather are unknown, so handling each task separately cannot deal with the complex practical scenarios. In this paper, we study the blind adverse weather removal problem. Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. The principle of MoE involves using adaptive networks to process different types of unknown inputs. Therefore, MoE has great potential for blind adverse weather removal. However, the original MoE module is inadequate for coupled multiple weather types and fails to utilize multi-scale features for better performance. To this end, we propose a method called Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. WM-MoE includes two key designs: WEather-Aware Router (WEAR) and Multi-Scale Experts (MSE). WEAR assigns experts for each image token based on decoupled content and weather features, which enhances the model's capability to process multiple adverse weathers. To obtain discriminative weather features from images, we propose Weather Guidance Fine-grained Contrastive Learning (WGF-CL), which utilizes weather cluster information to guide the assignment of positive and negative samples for each image token. Since processing different weather types requires different receptive fields, MSE leverages multi-scale features to enhance the spatial relationship modeling capability, facilitating the high-quality restoration of diverse weather types and intensities. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset. We also demonstrate the advantage of our method on downstream segmentation tasks.

Related papers

• SemiDDM-Weather: A Semi-supervised Learning Framework for All-in-one Adverse Weather Removal [57.52777076116241]
  Adverse weather removal aims to restore clear vision under adverse weather conditions. This paper presents a pioneering semi-supervised all-in-one adverse weather removal framework built on the teacher-student network.
  arXiv  Detail & Related papers  (2024-09-29T12:12:22Z)
• Multiple weather images restoration using the task transformer and adaptive mixup strategy [14.986500375481546]
  We introduce a novel multi-task severe weather removal model that can effectively handle complex weather conditions in an adaptive manner. Our model incorporates a weather task sequence generator, enabling the self-attention mechanism to selectively focus on features specific to different weather types. Our proposed model has achieved state-of-the-art performance on the publicly available dataset.
  arXiv  Detail & Related papers  (2024-09-05T04:55:40Z)
• Boosting Adverse Weather Crowd Counting via Multi-queue Contrastive Learning [8.692139673789555]
  We propose a two-stage crowd counting method to enhance the model's robustness in adverse weather. In the first stage, we introduce a multi-queue MoCo contrastive learning strategy to tackle the problem of weather class imbalance. In the second stage, we propose to refine the representations under the guidance of contrastive learning, enabling the conversion of the weather-aware representations to the normal weather domain.
  arXiv  Detail & Related papers  (2024-08-12T07:13:08Z)
• Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure [92.8834309803903]
  In real-world applications, image degeneration caused by adverse weather is always complex and changes with different weather conditions from days and seasons. We develop a novel continual learning framework with effective knowledge replay (KR) on a unified network structure. It considers the characteristics of the image restoration task with multiple degenerations in continual learning, and the knowledge for different degenerations can be shared and accumulated.
  arXiv  Detail & Related papers  (2024-03-12T03:50:57Z)
• Physical Perception Network and an All-weather Multi-modality Benchmark for Adverse Weather Image Fusion [4.3773535988950725]
  Multi-modality image fusion (MMIF) integrates the complementary information from different modal images to provide comprehensive and objective interpretation of a scenes. Existing MMIF methods lack the ability to resist different weather interferences in real-life scenarios. We have established a benchmark for MMIF research under extreme weather conditions.
  arXiv  Detail & Related papers  (2024-02-03T09:02:46Z)
• Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removal [8.58670633761819]
  All-in-one adverse weather removal is an emerging topic on image restoration, which aims to restore multiple weather degradations in an unified model. We propose a degradation type and severity aware model, called UtilityIR, for blind all-in-one bad weather image restoration.
  arXiv  Detail & Related papers  (2023-10-27T17:29:55Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• Counting Crowds in Bad Weather [68.50690406143173]
  We propose a method for robust crowd counting in adverse weather scenarios. Our model learns effective features and adaptive queries to account for large appearance variations. Experimental results show that the proposed algorithm is effective in counting crowds under different weather types on benchmark datasets.
  arXiv  Detail & Related papers  (2023-06-02T00:00:09Z)
• TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions [77.20136060506906]
  We propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder. TransWeather achieves significant improvements across multiple test datasets over both All-in-One network. It is validated on real world test images and found to be more effective than previous methods.
  arXiv  Detail & Related papers  (2021-11-29T18:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.