WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal

• URL: http://arxiv.org/abs/2303.13739v2
• Date: Thu, 4 Apr 2024 02:36:44 GMT
• Title: WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal
• Authors: Yulin Luo, Rui Zhao, Xiaobao Wei, Jinwei Chen, Yijie Lu, Shenghao Xie, Tianyu Wang, Ruiqin Xiong, Ming Lu, Shanghang Zhang,
• Abstract summary: Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. We propose Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset.
• Score: 38.257012295118145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adverse weather removal tasks like deraining, desnowing, and dehazing are usually treated as separate tasks. However, in practical autonomous driving scenarios, the type, intensity,and mixing degree of weather are unknown, so handling each task separately cannot deal with the complex practical scenarios. In this paper, we study the blind adverse weather removal problem. Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. The principle of MoE involves using adaptive networks to process different types of unknown inputs. Therefore, MoE has great potential for blind adverse weather removal. However, the original MoE module is inadequate for coupled multiple weather types and fails to utilize multi-scale features for better performance. To this end, we propose a method called Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. WM-MoE includes two key designs: WEather-Aware Router (WEAR) and Multi-Scale Experts (MSE). WEAR assigns experts for each image token based on decoupled content and weather features, which enhances the model's capability to process multiple adverse weathers. To obtain discriminative weather features from images, we propose Weather Guidance Fine-grained Contrastive Learning (WGF-CL), which utilizes weather cluster information to guide the assignment of positive and negative samples for each image token. Since processing different weather types requires different receptive fields, MSE leverages multi-scale features to enhance the spatial relationship modeling capability, facilitating the high-quality restoration of diverse weather types and intensities. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset. We also demonstrate the advantage of our method on downstream segmentation tasks.

Related papers

• MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers [44.600209414790854]
  Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. We propose a multi-weather Transformer, or MWFormer, that aims to solve multiple weather-induced degradations using a single architecture. We show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-26T08:47:39Z)
• WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning [69.82211470647349]
  We introduce the first generalist weather foundation model (WeatherGFM) It addresses a wide spectrum of weather understanding tasks in a unified manner. Our model can effectively handle up to ten weather understanding tasks, including weather forecasting, super-resolution, weather image translation, and post-processing.
  arXiv  Detail & Related papers  (2024-11-08T09:14:19Z)
• SemiDDM-Weather: A Semi-supervised Learning Framework for All-in-one Adverse Weather Removal [57.52777076116241]
  Adverse weather removal aims to restore clear vision under adverse weather conditions. This paper presents a pioneering semi-supervised all-in-one adverse weather removal framework built on the teacher-student network.
  arXiv  Detail & Related papers  (2024-09-29T12:12:22Z)
• Multiple weather images restoration using the task transformer and adaptive mixup strategy [14.986500375481546]
  We introduce a novel multi-task severe weather removal model that can effectively handle complex weather conditions in an adaptive manner. Our model incorporates a weather task sequence generator, enabling the self-attention mechanism to selectively focus on features specific to different weather types. Our proposed model has achieved state-of-the-art performance on the publicly available dataset.
  arXiv  Detail & Related papers  (2024-09-05T04:55:40Z)
• Boosting Adverse Weather Crowd Counting via Multi-queue Contrastive Learning [8.692139673789555]
  We propose a two-stage crowd counting method to enhance the model's robustness in adverse weather. In the first stage, we introduce a multi-queue MoCo contrastive learning strategy to tackle the problem of weather class imbalance. In the second stage, we propose to refine the representations under the guidance of contrastive learning, enabling the conversion of the weather-aware representations to the normal weather domain.
  arXiv  Detail & Related papers  (2024-08-12T07:13:08Z)
• Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure [92.8834309803903]
  In real-world applications, image degeneration caused by adverse weather is always complex and changes with different weather conditions from days and seasons. We develop a novel continual learning framework with effective knowledge replay (KR) on a unified network structure. It considers the characteristics of the image restoration task with multiple degenerations in continual learning, and the knowledge for different degenerations can be shared and accumulated.
  arXiv  Detail & Related papers  (2024-03-12T03:50:57Z)
• All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark [42.49073228252726]
  Multi-modality image fusion (MMIF) combines complementary information from different image modalities to provide a more comprehensive and objective interpretation of scenes. Existing MMIF methods lack the ability to resist different weather interferences in real-world scenes, preventing them from being useful in practical applications such as autonomous driving. We propose an all-weather MMIF model to achieve effective multi-tasking in this context. Experimental results in both real-world and synthetic scenes show that the proposed algorithm excels in detail recovery and multi-modality feature extraction.
  arXiv  Detail & Related papers  (2024-02-03T09:02:46Z)
• Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removal [8.58670633761819]
  All-in-one adverse weather removal is an emerging topic on image restoration, which aims to restore multiple weather degradations in an unified model. We propose a degradation type and severity aware model, called UtilityIR, for blind all-in-one bad weather image restoration.
  arXiv  Detail & Related papers  (2023-10-27T17:29:55Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• Counting Crowds in Bad Weather [68.50690406143173]
  We propose a method for robust crowd counting in adverse weather scenarios. Our model learns effective features and adaptive queries to account for large appearance variations. Experimental results show that the proposed algorithm is effective in counting crowds under different weather types on benchmark datasets.
  arXiv  Detail & Related papers  (2023-06-02T00:00:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.