Unsupervised Image Prior via Prompt Learning and CLIP Semantic Guidance for Low-Light Image Enhancement

• URL: http://arxiv.org/abs/2405.11478v1
• Date: Sun, 19 May 2024 08:06:14 GMT
• Title: Unsupervised Image Prior via Prompt Learning and CLIP Semantic Guidance for Low-Light Image Enhancement
• Authors: Igor Morawski, Kai He, Shusil Dangi, Winston H. Hsu,
• Abstract summary: We propose to improve the zero-reference low-light enhancement method by leveraging the rich visual-linguistic CLIP prior. We show that the proposed method leads to consistent improvements across various datasets regarding task-based performance.
• Score: 25.97198463881292
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Currently, low-light conditions present a significant challenge for machine cognition. In this paper, rather than optimizing models by assuming that human and machine cognition are correlated, we use zero-reference low-light enhancement to improve the performance of downstream task models. We propose to improve the zero-reference low-light enhancement method by leveraging the rich visual-linguistic CLIP prior without any need for paired or unpaired normal-light data, which is laborious and difficult to collect. We propose a simple but effective strategy to learn prompts that help guide the enhancement method and experimentally show that the prompts learned without any need for normal-light data improve image contrast, reduce over-enhancement, and reduce noise over-amplification. Next, we propose to reuse the CLIP model for semantic guidance via zero-shot open vocabulary classification to optimize low-light enhancement for task-based performance rather than human visual perception. We conduct extensive experimental results showing that the proposed method leads to consistent improvements across various datasets regarding task-based performance and compare our method against state-of-the-art methods, showing favorable results across various low-light datasets.

Related papers

• Unsupervised Low-light Image Enhancement with Lookup Tables and Diffusion Priors [38.96909959677438]
  Low-light image enhancement (LIE) aims at precisely and efficiently recovering an image degraded in poor illumination environments. Recent advanced LIE techniques are using deep neural networks, which require lots of low-normal light image pairs, network parameters, and computational resources. We devise a novel unsupervised LIE framework based on diffusion priors and lookup tables to achieve efficient low-light image recovery.
  arXiv  Detail & Related papers  (2024-09-27T16:37:27Z)
• Debiasing Multimodal Large Language Models [61.6896704217147]
  Large Vision-Language Models (LVLMs) have become indispensable tools in computer vision and natural language processing. Our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior to the input image. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)
• Visibility Enhancement for Low-light Hazy Scenarios [18.605784907840473]
  Low-light hazy scenes commonly appear at dusk and early morning. We propose a novel method to enhance visibility for low-light hazy scenarios. The framework is designed for enhancing visibility of the input image via fully utilizing the clues from different sub-tasks. The simulation is designed for generating the dataset with ground-truths by the proposed low-light hazy imaging model.
  arXiv  Detail & Related papers  (2023-08-01T15:07:38Z)
• Iterative Prompt Learning for Unsupervised Backlit Image Enhancement [86.90993077000789]
  We propose a novel unsupervised backlit image enhancement method, abbreviated as CLIP-LIT. We show that the open-world CLIP prior aids in distinguishing between backlit and well-lit images. Our method alternates between updating the prompt learning framework and enhancement network until visually pleasing results are achieved.
  arXiv  Detail & Related papers  (2023-03-30T17:37:14Z)
• Unified Vision and Language Prompt Learning [86.1530128487077]
  We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities.
  arXiv  Detail & Related papers  (2022-10-13T17:50:24Z)
• Enhancement by Your Aesthetic: An Intelligible Unsupervised Personalized Enhancer for Low-Light Images [67.14410374622699]
  We propose an intelligible unsupervised personalized enhancer (iUPEnhancer) for low-light images. The proposed iUP-Enhancer is trained with the guidance of these correlations and the corresponding unsupervised loss functions. Experiments demonstrate that the proposed algorithm produces competitive qualitative and quantitative results.
  arXiv  Detail & Related papers  (2022-07-15T07:16:10Z)
• Toward Fast, Flexible, and Robust Low-Light Image Enhancement [87.27326390675155]
  We develop a new Self-Calibrated Illumination (SCI) learning framework for fast, flexible, and robust brightening images in real-world low-light scenarios. Considering the computational burden of the cascaded pattern, we construct the self-calibrated module which realizes the convergence between results of each stage. We make comprehensive explorations to SCI's inherent properties including operation-insensitive adaptability and model-irrelevant generality.
  arXiv  Detail & Related papers  (2022-04-21T14:40:32Z)
• Semantic-Guided Zero-Shot Learning for Low-Light Image/Video Enhancement [3.4722706398428493]
  Low-light images challenge both human perceptions and computer vision algorithms. It is crucial to make algorithms robust to enlighten low-light images for computational photography and computer vision applications. This paper proposes a semantic-guided zero-shot low-light enhancement network which is trained in the absence of paired images.
  arXiv  Detail & Related papers  (2021-10-03T10:07:36Z)
• Improving Aerial Instance Segmentation in the Dark with Self-Supervised Low Light Enhancement [6.500738558466833]
  Low light conditions in aerial images adversely affect the performance of vision based applications. We propose a new method that is capable of enhancing the low light image in a self-supervised fashion. We also propose the generation of a new low light aerial dataset using GANs.
  arXiv  Detail & Related papers  (2021-02-10T12:24:40Z)
• Unsupervised Low-light Image Enhancement with Decoupled Networks [103.74355338972123]
  We learn a two-stage GAN-based framework to enhance the real-world low-light images in a fully unsupervised fashion. Our proposed method outperforms the state-of-the-art unsupervised image enhancement methods in terms of both illumination enhancement and noise reduction.
  arXiv  Detail & Related papers  (2020-05-06T13:37:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.