Language-driven All-in-one Adverse Weather Removal
- URL: http://arxiv.org/abs/2312.01381v1
- Date: Sun, 3 Dec 2023 13:05:54 GMT
- Title: Language-driven All-in-one Adverse Weather Removal
- Authors: Hao Yang, Liyuan Pan, Yan Yang, and Wei Liang
- Abstract summary: All-in-one (AiO) frameworks restore various adverse weather degradations with a single set of networks jointly.
Existing methods rely on extra supervision signals, which are usually unknown in real-world applications.
We propose a Language-driven Restoration framework (LDR) to alleviate the aforementioned issues.
- Score: 11.47695460133523
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: All-in-one (AiO) frameworks restore various adverse weather degradations with
a single set of networks jointly. To handle various weather conditions, an AiO
framework is expected to adaptively learn weather-specific knowledge for
different degradations and shared knowledge for common patterns. However,
existing methods: 1) rely on extra supervision signals, which are usually
unknown in real-world applications; 2) employ fixed network structures, which
restrict the diversity of weather-specific knowledge. In this paper, we propose
a Language-driven Restoration framework (LDR) to alleviate the aforementioned
issues. First, we leverage the power of pre-trained vision-language (PVL)
models to enrich the diversity of weather-specific knowledge by reasoning about
the occurrence, type, and severity of degradation, generating description-based
degradation priors. Then, with the guidance of degradation prior, we sparsely
select restoration experts from a candidate list dynamically based on a
Mixture-of-Experts (MoE) structure. This enables us to adaptively learn the
weather-specific and shared knowledge to handle various weather conditions
(e.g., unknown or mixed weather). Experiments on extensive restoration
scenarios show our superior performance (see Fig. 1). The source code will be
made available.
Related papers
- Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for
Video Adverse Weather Removal [53.15046196592023]
We introduce test-time adaptation into adverse weather removal in videos.
We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
arXiv Detail & Related papers (2024-03-12T14:21:30Z) - Continual All-in-One Adverse Weather Removal with Knowledge Replay on a
Unified Network Structure [92.8834309803903]
In real-world applications, image degeneration caused by adverse weather is always complex and changes with different weather conditions from days and seasons.
We develop a novel continual learning framework with effective knowledge replay (KR) on a unified network structure.
It considers the characteristics of the image restoration task with multiple degenerations in continual learning, and the knowledge for different degenerations can be shared and accumulated.
arXiv Detail & Related papers (2024-03-12T03:50:57Z) - Decoupling Degradation and Content Processing for Adverse Weather Image
Restoration [79.59228846484415]
Adverse weather image restoration strives to recover clear images from those affected by various weather types, such as rain, haze, and snow.
Previous techniques can handle multiple weather types within a single network, but they neglect the crucial distinction between these two processes, limiting the quality of restored images.
This work introduces a novel adverse weather image restoration method, called DDCNet, which decouples the degradation removal and content reconstruction process at the feature level based on their channel statistics.
arXiv Detail & Related papers (2023-12-08T12:26:38Z) - DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain
Question Answering over Knowledge Base and Text [73.68051228972024]
Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge.
Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
arXiv Detail & Related papers (2023-10-31T04:37:57Z) - Always Clear Days: Degradation Type and Severity Aware All-In-One
Adverse Weather Removal [8.58670633761819]
All-in-one adverse weather removal is an emerging topic on image restoration, which aims to restore multiple weather degradations in an unified model.
We propose a degradation type and severity aware model, called UtilityIR, for blind all-in-one bad weather image restoration.
arXiv Detail & Related papers (2023-10-27T17:29:55Z) - MetaWeather: Few-Shot Weather-Degraded Image Restoration [17.63266150036311]
We introduce MetaWeather, a universal approach that can handle diverse and novel weather conditions with a single unified model.
We show that MetaWeather can adapt to unseen weather conditions, significantly outperforming state-of-the-art multi-weather image restoration methods.
arXiv Detail & Related papers (2023-08-28T06:25:40Z) - Exploring the Application of Large-scale Pre-trained Models on Adverse
Weather Removal [97.53040662243768]
We propose a CLIP embedding module to make the network handle different weather conditions adaptively.
This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
arXiv Detail & Related papers (2023-06-15T10:06:13Z) - WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal [38.257012295118145]
Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks.
We propose Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal.
Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset.
arXiv Detail & Related papers (2023-03-24T01:46:25Z) - Weather GAN: Multi-Domain Weather Translation Using Generative
Adversarial Networks [76.64158017926381]
A new task is proposed, namely, weather translation, which refers to transferring weather conditions of the image from one category to another.
We develop a multi-domain weather translation approach based on generative adversarial networks (GAN), denoted as Weather GAN.
Our approach suppresses the distortion and deformation caused by weather translation.
arXiv Detail & Related papers (2021-03-09T13:51:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.