TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions

• URL: http://arxiv.org/abs/2111.14813v1
• Date: Mon, 29 Nov 2021 18:57:09 GMT
• Title: TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions
• Authors: Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M. Patel
• Abstract summary: We propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder. TransWeather achieves significant improvements across multiple test datasets over both All-in-One network. It is validated on real world test images and found to be more effective than previous methods.
• Score: 77.20136060506906
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Removing adverse weather conditions like rain, fog, and snow from images is an important problem in many applications. Most methods proposed in the literature have been designed to deal with just removing one type of degradation. Recently, a CNN-based method using neural architecture search (All-in-One) was proposed to remove all the weather conditions at once. However, it has a large number of parameters as it uses multiple encoders to cater to each weather removal task and still has scope for improvement in its performance. In this work, we focus on developing an efficient solution for the all adverse weather removal problem. To this end, we propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder that can restore an image degraded by any weather condition. Specifically, we utilize a novel transformer encoder using intra-patch transformer blocks to enhance attention inside the patches to effectively remove smaller weather degradations. We also introduce a transformer decoder with learnable weather type embeddings to adjust to the weather degradation at hand. TransWeather achieves significant improvements across multiple test datasets over both All-in-One network as well as methods fine-tuned for specific tasks. In particular, TransWeather pushes the current state-of-the-art by +6.34 PSNR on the Test1 (rain+fog) dataset, +4.93 PSNR on the SnowTest100K-L dataset and +3.11 PSNR on the RainDrop test dataset. TransWeather is also validated on real world test images and found to be more effective than previous methods. Implementation code and pre-trained weights can be accessed at https://github.com/jeya-maria-jose/TransWeather .

Related papers

• MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers [44.600209414790854]
  Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. We propose a multi-weather Transformer, or MWFormer, that aims to solve multiple weather-induced degradations using a single architecture. We show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-26T08:47:39Z)
• SemiDDM-Weather: A Semi-supervised Learning Framework for All-in-one Adverse Weather Removal [57.52777076116241]
  Adverse weather removal aims to restore clear vision under adverse weather conditions. This paper presents a pioneering semi-supervised all-in-one adverse weather removal framework built on the teacher-student network.
  arXiv  Detail & Related papers  (2024-09-29T12:12:22Z)
• Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal [53.15046196592023]
  We introduce test-time adaptation into adverse weather removal in videos. We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
  arXiv  Detail & Related papers  (2024-03-12T14:21:30Z)
• Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removal [8.58670633761819]
  All-in-one adverse weather removal is an emerging topic on image restoration, which aims to restore multiple weather degradations in an unified model. We propose a degradation type and severity aware model, called UtilityIR, for blind all-in-one bad weather image restoration.
  arXiv  Detail & Related papers  (2023-10-27T17:29:55Z)
• Learning Real-World Image De-Weathering with Imperfect Supervision [57.748585821252824]
  Existing real-world de-weathering datasets often exhibit inconsistent illumination, position, and textures between the ground-truth images and the input degraded images. We develop a Consistent Label Constructor (CLC) to generate a pseudo-label as consistent as possible with the input degraded image. We combine the original imperfect labels and pseudo-labels to jointly supervise the de-weathering model by the proposed Information Allocation Strategy.
  arXiv  Detail & Related papers  (2023-10-23T14:02:57Z)
• Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation [45.184188689391775]
  We propose the first framework for restoring videos from all adverse weather conditions by developing a video adverse-weather-component suppression network (ViWS-Net) Our ViWS-Net outperforms current state-of-the-art methods in terms of restoring videos degraded by any weather condition.
  arXiv  Detail & Related papers  (2023-09-24T17:13:55Z)
• Sit Back and Relax: Learning to Drive Incrementally in All Weather Conditions [16.014293219912]
  In autonomous driving scenarios, current object detection models show strong performance when tested in clear weather. We propose Domain-Incremental Learning through Activation Matching (DILAM) to adapt only the affine parameters of a clear weather pre-trained network to different weather conditions. Our memory bank is extremely lightweight, since affine parameters account for less than 2% of a typical object detector.
  arXiv  Detail & Related papers  (2023-05-30T11:37:41Z)
• ScatterNeRF: Seeing Through Fog with Physically-Based Inverse Neural Rendering [83.75284107397003]
  We introduce ScatterNeRF, a neural rendering method which renders scenes and decomposes the fog-free background. We propose a disentangled representation for the scattering volume and the scene objects, and learn the scene reconstruction with physics-inspired losses. We validate our method by capturing multi-view In-the-Wild data and controlled captures in a large-scale fog chamber.
  arXiv  Detail & Related papers  (2023-05-03T13:24:06Z)
• Towards an Effective and Efficient Transformer for Rain-by-snow Weather Removal [23.224536745724077]
  Rain-by-snow weather removal is a specialized task in weather-degraded image restoration aiming to eliminate coexisting rain streaks and snow particles. We propose RSFormer, an efficient and effective Transformer that addresses this challenge. RSFormer achieves the best trade-off between performance and time-consumption compared to other restoration methods.
  arXiv  Detail & Related papers  (2023-04-06T04:39:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.