Multiple weather images restoration using the task transformer and adaptive mixup strategy

• URL: http://arxiv.org/abs/2409.03249v1
• Date: Thu, 5 Sep 2024 04:55:40 GMT
• Title: Multiple weather images restoration using the task transformer and adaptive mixup strategy
• Authors: Yang Wen, Anyu Lai, Bo Qian, Hao Wang, Wuzhen Shi, Wenming Cao,
• Abstract summary: We introduce a novel multi-task severe weather removal model that can effectively handle complex weather conditions in an adaptive manner. Our model incorporates a weather task sequence generator, enabling the self-attention mechanism to selectively focus on features specific to different weather types. Our proposed model has achieved state-of-the-art performance on the publicly available dataset.
• Score: 14.986500375481546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The current state-of-the-art in severe weather removal predominantly focuses on single-task applications, such as rain removal, haze removal, and snow removal. However, real-world weather conditions often consist of a mixture of several weather types, and the degree of weather mixing in autonomous driving scenarios remains unknown. In the presence of complex and diverse weather conditions, a single weather removal model often encounters challenges in producing clear images from severe weather images. Therefore, there is a need for the development of multi-task severe weather removal models that can effectively handle mixed weather conditions and improve image quality in autonomous driving scenarios. In this paper, we introduce a novel multi-task severe weather removal model that can effectively handle complex weather conditions in an adaptive manner. Our model incorporates a weather task sequence generator, enabling the self-attention mechanism to selectively focus on features specific to different weather types. To tackle the challenge of repairing large areas of weather degradation, we introduce Fast Fourier Convolution (FFC) to increase the receptive field. Additionally, we propose an adaptive upsampling technique that effectively processes both the weather task information and underlying image features by selectively retaining relevant information. Our proposed model has achieved state-of-the-art performance on the publicly available dataset.

Related papers

• MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers [44.600209414790854]
  Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. We propose a multi-weather Transformer, or MWFormer, that aims to solve multiple weather-induced degradations using a single architecture. We show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-26T08:47:39Z)
• WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning [69.82211470647349]
  We introduce the first generalist weather foundation model (WeatherGFM) It addresses a wide spectrum of weather understanding tasks in a unified manner. Our model can effectively handle up to ten weather understanding tasks, including weather forecasting, super-resolution, weather image translation, and post-processing.
  arXiv  Detail & Related papers  (2024-11-08T09:14:19Z)
• Multi-Weather Image Restoration via Histogram-Based Transformer Feature Enhancement [14.986500375481546]
  In adverse weather conditions, single-weather restoration models struggle to meet practical demands. There is an urgent need for a model capable of effectively handling mixed weather conditions and enhancing image quality in an automated manner. We propose a Task Sequence Generator module that, in conjunction with the Task Intra-patch Block, effectively extracts task-specific features embedded in degraded images.
  arXiv  Detail & Related papers  (2024-09-10T08:47:03Z)
• Modeling Weather Uncertainty for Multi-weather Co-Presence Estimation [25.060597623607784]
  existing algorithms model weather condition as a discrete status and estimate it using multi-label classification. We consider the physical formulation of multi-weather conditions and model the impact of physical-related parameter on learning from the image appearance.
  arXiv  Detail & Related papers  (2024-03-29T10:05:29Z)
• Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure [92.8834309803903]
  In real-world applications, image degeneration caused by adverse weather is always complex and changes with different weather conditions from days and seasons. We develop a novel continual learning framework with effective knowledge replay (KR) on a unified network structure. It considers the characteristics of the image restoration task with multiple degenerations in continual learning, and the knowledge for different degenerations can be shared and accumulated.
  arXiv  Detail & Related papers  (2024-03-12T03:50:57Z)
• MetaWeather: Few-Shot Weather-Degraded Image Restoration [17.63266150036311]
  We introduce MetaWeather, a universal approach that can handle diverse and novel weather conditions with a single unified model. We show that MetaWeather can adapt to unseen weather conditions, significantly outperforming state-of-the-art multi-weather image restoration methods.
  arXiv  Detail & Related papers  (2023-08-28T06:25:40Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• Counting Crowds in Bad Weather [68.50690406143173]
  We propose a method for robust crowd counting in adverse weather scenarios. Our model learns effective features and adaptive queries to account for large appearance variations. Experimental results show that the proposed algorithm is effective in counting crowds under different weather types on benchmark datasets.
  arXiv  Detail & Related papers  (2023-06-02T00:00:09Z)
• ScatterNeRF: Seeing Through Fog with Physically-Based Inverse Neural Rendering [83.75284107397003]
  We introduce ScatterNeRF, a neural rendering method which renders scenes and decomposes the fog-free background. We propose a disentangled representation for the scattering volume and the scene objects, and learn the scene reconstruction with physics-inspired losses. We validate our method by capturing multi-view In-the-Wild data and controlled captures in a large-scale fog chamber.
  arXiv  Detail & Related papers  (2023-05-03T13:24:06Z)
• WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal [38.257012295118145]
  Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. We propose Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset.
  arXiv  Detail & Related papers  (2023-03-24T01:46:25Z)
• TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions [77.20136060506906]
  We propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder. TransWeather achieves significant improvements across multiple test datasets over both All-in-One network. It is validated on real world test images and found to be more effective than previous methods.
  arXiv  Detail & Related papers  (2021-11-29T18:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.