Underwater Image Enhancement by Diffusion Model with Customized CLIP-Classifier
- URL: http://arxiv.org/abs/2405.16214v2
- Date: Fri, 7 Jun 2024 09:07:18 GMT
- Title: Underwater Image Enhancement by Diffusion Model with Customized CLIP-Classifier
- Authors: Shuaixin Liu, Kunqian Li, Yilin Ding, Qi Qi,
- Abstract summary: Underwater Image Enhancement (UIE) aims to improve the visual quality from a low-quality input.
We propose CLIP-UIE, a novel framework that leverages the potential of Contrastive Language-Image Pretraining (CLIP) for the UIE task.
Specifically, we propose employing color transfer to yield synthetic images by degrading in-air natural images into corresponding underwater images, guided by the real underwater domain.
- Score: 5.352081564604589
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Underwater Image Enhancement (UIE) aims to improve the visual quality from a low-quality input. Unlike other image enhancement tasks, underwater images suffer from the unavailability of real reference images. Although existing works exploit synthetic images and manually select well-enhanced images as reference images to train enhancement networks, their upper performance bound is limited by the reference domain. To address this challenge, we propose CLIP-UIE, a novel framework that leverages the potential of Contrastive Language-Image Pretraining (CLIP) for the UIE task. Specifically, we propose employing color transfer to yield synthetic images by degrading in-air natural images into corresponding underwater images, guided by the real underwater domain. This approach enables the diffusion model to capture the prior knowledge of mapping transitions from the underwater degradation domain to the real in-air natural domain. Still, fine-tuning the diffusion model for specific downstream tasks is inevitable and may result in the loss of this prior knowledge. To migrate this drawback, we combine the prior knowledge of the in-air natural domain with CLIP to train a CLIP-Classifier. Subsequently, we integrate this CLIP-Classifier with UIE benchmark datasets to jointly fine-tune the diffusion model, guiding the enhancement results towards the in-air natural domain. Additionally, for image enhancement tasks, we observe that both the image-to-image diffusion model and CLIP-Classifier primarily focus on the high-frequency region during fine-tuning. Therefore, we propose a new fine-tuning strategy that specifically targets the high-frequency region, which can be up to 10 times faster than traditional strategies. Extensive experiments demonstrate that our method exhibits a more natural appearance.
Related papers
- HazeCLIP: Towards Language Guided Real-World Image Dehazing [62.4454483961341]
Existing methods have achieved remarkable performance in single image dehazing, particularly on synthetic datasets.
This paper introduces HazeCLIP, a language-guided adaptation framework designed to enhance the real-world performance of pre-trained dehazing networks.
arXiv Detail & Related papers (2024-07-18T17:18:25Z) - DP-IQA: Utilizing Diffusion Prior for Blind Image Quality Assessment in the Wild [54.139923409101044]
We propose a novel IQA method called diffusion priors-based IQA (DP-IQA)
We use pre-trained stable diffusion as the backbone, extract multi-level features from the denoising U-Net, and decode them to estimate the image quality score.
We distill the knowledge in the above model into a CNN-based student model, significantly reducing the parameter to enhance applicability.
arXiv Detail & Related papers (2024-05-30T12:32:35Z) - Active Generation for Image Classification [50.18107721267218]
We propose to address the efficiency of image generation by focusing on the specific needs and characteristics of the model.
With a central tenet of active learning, our method, named ActGen, takes a training-aware approach to image generation.
arXiv Detail & Related papers (2024-03-11T08:45:31Z) - DGNet: Dynamic Gradient-Guided Network for Water-Related Optics Image
Enhancement [77.0360085530701]
Underwater image enhancement (UIE) is a challenging task due to the complex degradation caused by underwater environments.
Previous methods often idealize the degradation process, and neglect the impact of medium noise and object motion on the distribution of image features.
Our approach utilizes predicted images to dynamically update pseudo-labels, adding a dynamic gradient to optimize the network's gradient space.
arXiv Detail & Related papers (2023-12-12T06:07:21Z) - UIEDP:Underwater Image Enhancement with Diffusion Prior [20.349103580702028]
Underwater image enhancement (UIE) aims to generate clear images from low-quality underwater images.
We propose UIEDP, a novel framework treating UIE as a posterior distribution sampling process of clear images conditioned on degraded underwater inputs.
arXiv Detail & Related papers (2023-12-11T09:24:52Z) - LLDiffusion: Learning Degradation Representations in Diffusion Models
for Low-Light Image Enhancement [118.83316133601319]
Current deep learning methods for low-light image enhancement (LLIE) typically rely on pixel-wise mapping learned from paired data.
We propose a degradation-aware learning scheme for LLIE using diffusion models, which effectively integrates degradation and image priors into the diffusion process.
arXiv Detail & Related papers (2023-07-27T07:22:51Z) - PUGAN: Physical Model-Guided Underwater Image Enhancement Using GAN with
Dual-Discriminators [120.06891448820447]
How to obtain clear and visually pleasant images has become a common concern of people.
The task of underwater image enhancement (UIE) has also emerged as the times require.
In this paper, we propose a physical model-guided GAN model for UIE, referred to as PUGAN.
Our PUGAN outperforms state-of-the-art methods in both qualitative and quantitative metrics.
arXiv Detail & Related papers (2023-06-15T07:41:12Z) - Domain Adaptation for Underwater Image Enhancement via Content and Style
Separation [7.077978580799124]
Underwater image suffer from color cast, low contrast and hazy effect due to light absorption, refraction and scattering.
Recent learning-based methods demonstrate astonishing performance on underwater image enhancement.
We propose a domain adaptation framework for underwater image enhancement via content and style separation.
arXiv Detail & Related papers (2022-02-17T09:30:29Z) - Wavelength-based Attributed Deep Neural Network for Underwater Image
Restoration [9.378355457555319]
This paper shows that attributing the right receptive field size (context) based on the traversing range of the color channel may lead to a substantial performance gain.
As a second novelty, we have incorporated an attentive skip mechanism to adaptively refine the learned multi-contextual features.
The proposed framework, called Deep WaveNet, is optimized using the traditional pixel-wise and feature-based cost functions.
arXiv Detail & Related papers (2021-06-15T06:47:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.