Language-driven All-in-one Adverse Weather Removal

• URL: http://arxiv.org/abs/2312.01381v1
• Date: Sun, 3 Dec 2023 13:05:54 GMT
• Title: Language-driven All-in-one Adverse Weather Removal
• Authors: Hao Yang, Liyuan Pan, Yan Yang, and Wei Liang
• Abstract summary: All-in-one (AiO) frameworks restore various adverse weather degradations with a single set of networks jointly. Existing methods rely on extra supervision signals, which are usually unknown in real-world applications. We propose a Language-driven Restoration framework (LDR) to alleviate the aforementioned issues.
• Score: 11.47695460133523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: All-in-one (AiO) frameworks restore various adverse weather degradations with a single set of networks jointly. To handle various weather conditions, an AiO framework is expected to adaptively learn weather-specific knowledge for different degradations and shared knowledge for common patterns. However, existing methods: 1) rely on extra supervision signals, which are usually unknown in real-world applications; 2) employ fixed network structures, which restrict the diversity of weather-specific knowledge. In this paper, we propose a Language-driven Restoration framework (LDR) to alleviate the aforementioned issues. First, we leverage the power of pre-trained vision-language (PVL) models to enrich the diversity of weather-specific knowledge by reasoning about the occurrence, type, and severity of degradation, generating description-based degradation priors. Then, with the guidance of degradation prior, we sparsely select restoration experts from a candidate list dynamically based on a Mixture-of-Experts (MoE) structure. This enables us to adaptively learn the weather-specific and shared knowledge to handle various weather conditions (e.g., unknown or mixed weather). Experiments on extensive restoration scenarios show our superior performance (see Fig. 1). The source code will be made available.

Related papers

• WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning [69.82211470647349]
  We introduce the first generalist weather foundation model (WeatherGFM) It addresses a wide spectrum of weather understanding tasks in a unified manner. Our model can effectively handle up to ten weather understanding tasks, including weather forecasting, super-resolution, weather image translation, and post-processing.
  arXiv  Detail & Related papers  (2024-11-08T09:14:19Z)
• A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends [67.43992456058541]
  Image restoration (IR) refers to the process of improving visual quality of images while removing degradation, such as noise, blur, weather effects, and so on. Traditional IR methods typically target specific types of degradation, which limits their effectiveness in real-world scenarios with complex distortions. The all-in-one image restoration (AiOIR) paradigm has emerged, offering a unified framework that adeptly addresses multiple degradation types.
  arXiv  Detail & Related papers  (2024-10-19T11:11:09Z)
• Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraint [15.733168323227174]
  We introduce a novel pipeline, T3-DiffWeather, to handle unpredictable weather input. We employ a prompt pool that allows the network to autonomously combine sub-prompts to construct weather-prompts. Our method achieves state-of-the-art performance across various synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-09-24T04:46:18Z)
• Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal [53.15046196592023]
  We introduce test-time adaptation into adverse weather removal in videos. We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
  arXiv  Detail & Related papers  (2024-03-12T14:21:30Z)
• Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure [92.8834309803903]
  In real-world applications, image degeneration caused by adverse weather is always complex and changes with different weather conditions from days and seasons. We develop a novel continual learning framework with effective knowledge replay (KR) on a unified network structure. It considers the characteristics of the image restoration task with multiple degenerations in continual learning, and the knowledge for different degenerations can be shared and accumulated.
  arXiv  Detail & Related papers  (2024-03-12T03:50:57Z)
• Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removal [8.58670633761819]
  All-in-one adverse weather removal is an emerging topic on image restoration, which aims to restore multiple weather degradations in an unified model. We propose a degradation type and severity aware model, called UtilityIR, for blind all-in-one bad weather image restoration.
  arXiv  Detail & Related papers  (2023-10-27T17:29:55Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• Weather GAN: Multi-Domain Weather Translation Using Generative Adversarial Networks [76.64158017926381]
  A new task is proposed, namely, weather translation, which refers to transferring weather conditions of the image from one category to another. We develop a multi-domain weather translation approach based on generative adversarial networks (GAN), denoted as Weather GAN. Our approach suppresses the distortion and deformation caused by weather translation.
  arXiv  Detail & Related papers  (2021-03-09T13:51:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.