WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal
- URL: http://arxiv.org/abs/2303.13739v2
- Date: Thu, 4 Apr 2024 02:36:44 GMT
- Title: WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal
- Authors: Yulin Luo, Rui Zhao, Xiaobao Wei, Jinwei Chen, Yijie Lu, Shenghao Xie, Tianyu Wang, Ruiqin Xiong, Ming Lu, Shanghang Zhang,
- Abstract summary: Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks.
We propose Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal.
Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset.
- Score: 38.257012295118145
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Adverse weather removal tasks like deraining, desnowing, and dehazing are usually treated as separate tasks. However, in practical autonomous driving scenarios, the type, intensity,and mixing degree of weather are unknown, so handling each task separately cannot deal with the complex practical scenarios. In this paper, we study the blind adverse weather removal problem. Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. The principle of MoE involves using adaptive networks to process different types of unknown inputs. Therefore, MoE has great potential for blind adverse weather removal. However, the original MoE module is inadequate for coupled multiple weather types and fails to utilize multi-scale features for better performance. To this end, we propose a method called Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. WM-MoE includes two key designs: WEather-Aware Router (WEAR) and Multi-Scale Experts (MSE). WEAR assigns experts for each image token based on decoupled content and weather features, which enhances the model's capability to process multiple adverse weathers. To obtain discriminative weather features from images, we propose Weather Guidance Fine-grained Contrastive Learning (WGF-CL), which utilizes weather cluster information to guide the assignment of positive and negative samples for each image token. Since processing different weather types requires different receptive fields, MSE leverages multi-scale features to enhance the spatial relationship modeling capability, facilitating the high-quality restoration of diverse weather types and intensities. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset. We also demonstrate the advantage of our method on downstream segmentation tasks.
Related papers
- Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for
Video Adverse Weather Removal [53.15046196592023]
We introduce test-time adaptation into adverse weather removal in videos.
We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
arXiv Detail & Related papers (2024-03-12T14:21:30Z) - Continual All-in-One Adverse Weather Removal with Knowledge Replay on a
Unified Network Structure [92.8834309803903]
In real-world applications, image degeneration caused by adverse weather is always complex and changes with different weather conditions from days and seasons.
We develop a novel continual learning framework with effective knowledge replay (KR) on a unified network structure.
It considers the characteristics of the image restoration task with multiple degenerations in continual learning, and the knowledge for different degenerations can be shared and accumulated.
arXiv Detail & Related papers (2024-03-12T03:50:57Z) - Physical Perception Network and an All-weather Multi-modality Benchmark
for Adverse Weather Image Fusion [4.3773535988950725]
Multi-modality image fusion (MMIF) integrates the complementary information from different modal images to provide comprehensive and objective interpretation of a scenes.
Existing MMIF methods lack the ability to resist different weather interferences in real-life scenarios.
We have established a benchmark for MMIF research under extreme weather conditions.
arXiv Detail & Related papers (2024-02-03T09:02:46Z) - Always Clear Days: Degradation Type and Severity Aware All-In-One
Adverse Weather Removal [8.58670633761819]
All-in-one adverse weather removal is an emerging topic on image restoration, which aims to restore multiple weather degradations in an unified model.
We propose a degradation type and severity aware model, called UtilityIR, for blind all-in-one bad weather image restoration.
arXiv Detail & Related papers (2023-10-27T17:29:55Z) - Exploring the Application of Large-scale Pre-trained Models on Adverse
Weather Removal [97.53040662243768]
We propose a CLIP embedding module to make the network handle different weather conditions adaptively.
This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
arXiv Detail & Related papers (2023-06-15T10:06:13Z) - Counting Crowds in Bad Weather [68.50690406143173]
We propose a method for robust crowd counting in adverse weather scenarios.
Our model learns effective features and adaptive queries to account for large appearance variations.
Experimental results show that the proposed algorithm is effective in counting crowds under different weather types on benchmark datasets.
arXiv Detail & Related papers (2023-06-02T00:00:09Z) - TransWeather: Transformer-based Restoration of Images Degraded by
Adverse Weather Conditions [77.20136060506906]
We propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder.
TransWeather achieves significant improvements across multiple test datasets over both All-in-One network.
It is validated on real world test images and found to be more effective than previous methods.
arXiv Detail & Related papers (2021-11-29T18:57:09Z) - RiWNet: A moving object instance segmentation Network being Robust in
adverse Weather conditions [13.272209740926156]
We focus on a new possibility, that is, to improve its resilience to weather interference through the network's structural design.
We propose a novel FPN structure called RiWFPN with a progressive top-down interaction and attention refinement module.
We extend SOLOV2 to capture temporal information in video to learn motion information, and propose a moving object instance segmentation network with RiWFPN called RiWNet.
arXiv Detail & Related papers (2021-09-04T08:55:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.