Underwater Image Enhancement by Diffusion Model with Customized CLIP-Classifier

• URL: http://arxiv.org/abs/2405.16214v2
• Date: Fri, 7 Jun 2024 09:07:18 GMT
• Title: Underwater Image Enhancement by Diffusion Model with Customized CLIP-Classifier
• Authors: Shuaixin Liu, Kunqian Li, Yilin Ding, Qi Qi,
• Abstract summary: Underwater Image Enhancement (UIE) aims to improve the visual quality from a low-quality input. We propose CLIP-UIE, a novel framework that leverages the potential of Contrastive Language-Image Pretraining (CLIP) for the UIE task. Specifically, we propose employing color transfer to yield synthetic images by degrading in-air natural images into corresponding underwater images, guided by the real underwater domain.
• Score: 5.352081564604589
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Underwater Image Enhancement (UIE) aims to improve the visual quality from a low-quality input. Unlike other image enhancement tasks, underwater images suffer from the unavailability of real reference images. Although existing works exploit synthetic images and manually select well-enhanced images as reference images to train enhancement networks, their upper performance bound is limited by the reference domain. To address this challenge, we propose CLIP-UIE, a novel framework that leverages the potential of Contrastive Language-Image Pretraining (CLIP) for the UIE task. Specifically, we propose employing color transfer to yield synthetic images by degrading in-air natural images into corresponding underwater images, guided by the real underwater domain. This approach enables the diffusion model to capture the prior knowledge of mapping transitions from the underwater degradation domain to the real in-air natural domain. Still, fine-tuning the diffusion model for specific downstream tasks is inevitable and may result in the loss of this prior knowledge. To migrate this drawback, we combine the prior knowledge of the in-air natural domain with CLIP to train a CLIP-Classifier. Subsequently, we integrate this CLIP-Classifier with UIE benchmark datasets to jointly fine-tune the diffusion model, guiding the enhancement results towards the in-air natural domain. Additionally, for image enhancement tasks, we observe that both the image-to-image diffusion model and CLIP-Classifier primarily focus on the high-frequency region during fine-tuning. Therefore, we propose a new fine-tuning strategy that specifically targets the high-frequency region, which can be up to 10 times faster than traditional strategies. Extensive experiments demonstrate that our method exhibits a more natural appearance.

Related papers

• Towards Effective User Attribution for Latent Diffusion Models via Watermark-Informed Blending [54.26862913139299]
  We introduce a novel framework Towards Effective user Attribution for latent diffusion models via Watermark-Informed Blending (TEAWIB) TEAWIB incorporates a unique ready-to-use configuration approach that allows seamless integration of user-specific watermarks into generative models. Experiments validate the effectiveness of TEAWIB, showcasing the state-of-the-art performance in perceptual quality and attribution accuracy.
  arXiv  Detail & Related papers  (2024-09-17T07:52:09Z)
• UIE-UnFold: Deep Unfolding Network with Color Priors and Vision Transformer for Underwater Image Enhancement [27.535028176427623]
  Underwater image enhancement (UIE) plays a crucial role in various marine applications. Current learning-based approaches frequently lack explicit prior knowledge about the physical processes involved in underwater image formation. This paper proposes a novel deep unfolding network (DUN) for UIE that integrates color priors and inter-stage feature incorporation.
  arXiv  Detail & Related papers  (2024-08-20T08:48:33Z)
• HazeCLIP: Towards Language Guided Real-World Image Dehazing [62.4454483961341]
  Existing methods have achieved remarkable performance in single image dehazing, particularly on synthetic datasets. This paper introduces HazeCLIP, a language-guided adaptation framework designed to enhance the real-world performance of pre-trained dehazing networks.
  arXiv  Detail & Related papers  (2024-07-18T17:18:25Z)
• DGNet: Dynamic Gradient-Guided Network for Water-Related Optics Image Enhancement [77.0360085530701]
  Underwater image enhancement (UIE) is a challenging task due to the complex degradation caused by underwater environments. Previous methods often idealize the degradation process, and neglect the impact of medium noise and object motion on the distribution of image features. Our approach utilizes predicted images to dynamically update pseudo-labels, adding a dynamic gradient to optimize the network's gradient space.
  arXiv  Detail & Related papers  (2023-12-12T06:07:21Z)
• UIEDP:Underwater Image Enhancement with Diffusion Prior [20.349103580702028]
  Underwater image enhancement (UIE) aims to generate clear images from low-quality underwater images. We propose UIEDP, a novel framework treating UIE as a posterior distribution sampling process of clear images conditioned on degraded underwater inputs.
  arXiv  Detail & Related papers  (2023-12-11T09:24:52Z)
• LLDiffusion: Learning Degradation Representations in Diffusion Models for Low-Light Image Enhancement [118.83316133601319]
  Current deep learning methods for low-light image enhancement (LLIE) typically rely on pixel-wise mapping learned from paired data. We propose a degradation-aware learning scheme for LLIE using diffusion models, which effectively integrates degradation and image priors into the diffusion process.
  arXiv  Detail & Related papers  (2023-07-27T07:22:51Z)
• PUGAN: Physical Model-Guided Underwater Image Enhancement Using GAN with Dual-Discriminators [120.06891448820447]
  How to obtain clear and visually pleasant images has become a common concern of people. The task of underwater image enhancement (UIE) has also emerged as the times require. In this paper, we propose a physical model-guided GAN model for UIE, referred to as PUGAN. Our PUGAN outperforms state-of-the-art methods in both qualitative and quantitative metrics.
  arXiv  Detail & Related papers  (2023-06-15T07:41:12Z)
• Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images [60.34381768479834]
  Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language. We pioneer a systematic study on deepfake detection generated by state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-04-02T10:25:09Z)
• Domain Adaptation for Underwater Image Enhancement via Content and Style Separation [7.077978580799124]
  Underwater image suffer from color cast, low contrast and hazy effect due to light absorption, refraction and scattering. Recent learning-based methods demonstrate astonishing performance on underwater image enhancement. We propose a domain adaptation framework for underwater image enhancement via content and style separation.
  arXiv  Detail & Related papers  (2022-02-17T09:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.