Transfer CLIP for Generalizable Image Denoising

• URL: http://arxiv.org/abs/2403.15132v1
• Date: Fri, 22 Mar 2024 11:33:04 GMT
• Title: Transfer CLIP for Generalizable Image Denoising
• Authors: Jun Cheng, Dong Liang, Shan Tan,
• Abstract summary: We devise an asymmetrical encoder-decoder denoising network, which incorporates dense features including the noisy image. Experiments and comparisons conducted across diverse OOD noises, including synthetic noise, real-world sRGB noise, and low-dose CT image noise, demonstrate the superior generalization ability of our method.
• Score: 11.144858989063522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image denoising is a fundamental task in computer vision. While prevailing deep learning-based supervised and self-supervised methods have excelled in eliminating in-distribution noise, their susceptibility to out-of-distribution (OOD) noise remains a significant challenge. The recent emergence of contrastive language-image pre-training (CLIP) model has showcased exceptional capabilities in open-world image recognition and segmentation. Yet, the potential for leveraging CLIP to enhance the robustness of low-level tasks remains largely unexplored. This paper uncovers that certain dense features extracted from the frozen ResNet image encoder of CLIP exhibit distortion-invariant and content-related properties, which are highly desirable for generalizable denoising. Leveraging these properties, we devise an asymmetrical encoder-decoder denoising network, which incorporates dense features including the noisy image and its multi-scale features from the frozen ResNet encoder of CLIP into a learnable image decoder to achieve generalizable denoising. The progressive feature augmentation strategy is further proposed to mitigate feature overfitting and improve the robustness of the learnable decoder. Extensive experiments and comparisons conducted across diverse OOD noises, including synthetic noise, real-world sRGB noise, and low-dose CT image noise, demonstrate the superior generalization ability of our method.

Related papers

• Joint End-to-End Image Compression and Denoising: Leveraging Contrastive Learning and Multi-Scale Self-ONNs [18.71504105967766]
  Noisy images are a challenge to image compression algorithms due to the inherent difficulty of compressing noise. We propose a novel method integrating a multi-scale denoiser comprising of Self Organizing Operational Neural Networks, for joint image compression and denoising.
  arXiv  Detail & Related papers  (2024-02-08T11:33:16Z)
• Advancing Unsupervised Low-light Image Enhancement: Noise Estimation, Illumination Interpolation, and Self-Regulation [55.07472635587852]
  Low-Light Image Enhancement (LLIE) techniques have made notable advancements in preserving image details and enhancing contrast. These approaches encounter persistent challenges in efficiently mitigating dynamic noise and accommodating diverse low-light scenarios. We first propose a method for estimating the noise level in low light images in a quick and accurate way. We then devise a Learnable Illumination Interpolator (LII) to satisfy general constraints between illumination and input.
  arXiv  Detail & Related papers  (2023-05-17T13:56:48Z)
• CurvPnP: Plug-and-play Blind Image Restoration with Deep Curvature Denoiser [7.442030347967277]
  existing plug-and-play image restoration methods are designed for non-blind denoising. We propose a novel framework with blind prior, which can deal with more complicated image restoration problems in the real world. Our model is shown to be able to recover the fine image details tiny structures even when the noise level is different.
  arXiv  Detail & Related papers  (2022-11-14T11:30:24Z)
• Deep Semantic Statistics Matching (D2SM) Denoising Network [70.01091467628068]
  We introduce the Deep Semantic Statistics Matching (D2SM) Denoising Network. It exploits semantic features of pretrained classification networks, then it implicitly matches the probabilistic distribution of clear images at the semantic feature space. By learning to preserve the semantic distribution of denoised images, we empirically find our method significantly improves the denoising capabilities of networks.
  arXiv  Detail & Related papers  (2022-07-19T14:35:42Z)
• Zero-shot Blind Image Denoising via Implicit Neural Representations [77.79032012459243]
  We propose an alternative denoising strategy that leverages the architectural inductive bias of implicit neural representations (INRs) We show that our method outperforms existing zero-shot denoising methods under an extensive set of low-noise or real-noise scenarios.
  arXiv  Detail & Related papers  (2022-04-05T12:46:36Z)
• Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis [148.16279746287452]
  We propose a swin-conv block to incorporate the local modeling ability of residual convolutional layer and non-local modeling ability of swin transformer block. For the training data synthesis, we design a practical noise degradation model which takes into consideration different kinds of noise. Experiments on AGWN removal and real image denoising demonstrate that the new network architecture design achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-03-24T18:11:31Z)
• Reconstructing the Noise Manifold for Image Denoising [56.562855317536396]
  We introduce the idea of a cGAN which explicitly leverages structure in the image noise space. By learning directly a low dimensional manifold of the image noise, the generator promotes the removal from the noisy image only that information which spans this manifold. Based on our experiments, our model substantially outperforms existing state-of-the-art architectures.
  arXiv  Detail & Related papers  (2020-02-11T00:31:31Z)
• Variational Denoising Network: Toward Blind Noise Modeling and Removal [59.36166491196973]
  Blind image denoising is an important yet very challenging problem in computer vision. We propose a new variational inference method, which integrates both noise estimation and image denoising.
  arXiv  Detail & Related papers  (2019-08-29T15:54:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.