Towards Real-World Adverse Weather Image Restoration: Enhancing Clearness and Semantics with Vision-Language Models

• URL: http://arxiv.org/abs/2409.02101v1
• Date: Tue, 3 Sep 2024 17:56:51 GMT
• Title: Towards Real-World Adverse Weather Image Restoration: Enhancing Clearness and Semantics with Vision-Language Models
• Authors: Jiaqi Xu, Mengyang Wu, Xiaowei Hu, Chi-Wing Fu, Qi Dou, Pheng-Ann Heng,
• Abstract summary: We formulate a semi-supervised learning framework employing vision-language models to enhance restoration performance in real-world conditions. For clearness enhancement, we use real-world data, utilizing a dual-step strategy with pseudo-labels assessed by vision-language models and weather prompt learning. For semantic enhancement, we integrate real-world data by adjusting weather conditions in vision-language model descriptions while preserving semantic meaning.
• Score: 85.53913950137175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper addresses the limitations of adverse weather image restoration approaches trained on synthetic data when applied to real-world scenarios. We formulate a semi-supervised learning framework employing vision-language models to enhance restoration performance across diverse adverse weather conditions in real-world settings. Our approach involves assessing image clearness and providing semantics using vision-language models on real data, serving as supervision signals for training restoration models. For clearness enhancement, we use real-world data, utilizing a dual-step strategy with pseudo-labels assessed by vision-language models and weather prompt learning. For semantic enhancement, we integrate real-world data by adjusting weather conditions in vision-language model descriptions while preserving semantic meaning. Additionally, we introduce an effective training strategy to bootstrap restoration performance. Our approach achieves superior results in real-world adverse weather image restoration, demonstrated through qualitative and quantitative comparisons with state-of-the-art works.

Related papers

• Zero-Shot Object-Centric Representation Learning [72.43369950684057]
  We study current object-centric methods through the lens of zero-shot generalization. We introduce a benchmark comprising eight different synthetic and real-world datasets. We find that training on diverse real-world images improves transferability to unseen scenarios.
  arXiv  Detail & Related papers  (2024-08-17T10:37:07Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather [9.619700283574533]
  We introduce a general paired-training method that leads to improved performance on images in adverse weather conditions. We create the first semantic segmentation dataset with accurate clear and adverse weather image pairs. We find that training on these paired clear and adverse weather frames which share an underlying scene results in improved performance on adverse weather data.
  arXiv  Detail & Related papers  (2023-12-15T04:57:54Z)
• ReCoRe: Regularized Contrastive Representation Learning of World Model [21.29132219042405]
  We present a world model that learns invariant features using contrastive unsupervised learning and an intervention-invariant regularizer. Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark.
  arXiv  Detail & Related papers  (2023-12-14T15:53:07Z)
• WeatherDepth: Curriculum Contrastive Learning for Self-Supervised Depth Estimation under Adverse Weather Conditions [42.99525455786019]
  We propose WeatherDepth, a self-supervised robust depth estimation model with curriculum contrastive learning. The proposed solution is proven to be easily incorporated into various architectures and demonstrates state-of-the-art (SoTA) performance on both synthetic and real weather datasets.
  arXiv  Detail & Related papers  (2023-10-09T09:26:27Z)
• Perceptual Grouping in Contrastive Vision-Language Models [59.1542019031645]
  We show how vision-language models are able to understand where objects reside within an image and group together visually related parts of the imagery. We propose a minimal set of modifications that results in models that uniquely learn both semantic and spatial information.
  arXiv  Detail & Related papers  (2022-10-18T17:01:35Z)
• Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models [8.122270502556374]
  We present a novel patch-based image restoration algorithm based on denoising diffusion probabilistic models. We demonstrate our approach to achieve state-of-the-art performances on both weather-specific and multi-weather image restoration.
  arXiv  Detail & Related papers  (2022-07-29T11:52:41Z)
• Unsupervised Restoration of Weather-affected Images using Deep Gaussian Process-based CycleGAN [92.15895515035795]
  We describe an approach for supervising deep networks that are based on CycleGAN. We introduce new losses for training CycleGAN that lead to more effective training, resulting in high-quality reconstructions. We demonstrate that the proposed method can be effectively applied to different restoration tasks like de-raining, de-hazing and de-snowing.
  arXiv  Detail & Related papers  (2022-04-23T01:30:47Z)
• Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement [78.58603635621591]
  Training an unpaired synthetic-to-real translation network in image space is severely under-constrained. We propose a semi-supervised approach that operates on the disentangled shading and albedo layers of the image. Our two-stage pipeline first learns to predict accurate shading in a supervised fashion using physically-based renderings as targets.
  arXiv  Detail & Related papers  (2020-03-27T21:45:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.