Boosting Image Restoration via Priors from Pre-trained Models

• URL: http://arxiv.org/abs/2403.06793v2
• Date: Tue, 19 Mar 2024 04:46:42 GMT
• Title: Boosting Image Restoration via Priors from Pre-trained Models
• Authors: Xiaogang Xu, Shu Kong, Tao Hu, Zhe Liu, Hujun Bao,
• Abstract summary: We learn an additional lightweight module called Pre-Train-Guided Refinement Module (PTG-RM) to refine restoration results of a target restoration network with OSF. PTG-RM effectively enhances restoration performance of various models across different tasks, including low-light enhancement, deraining, deblurring, and denoising.
• Score: 54.83907596825985
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained models with large-scale training data, such as CLIP and Stable Diffusion, have demonstrated remarkable performance in various high-level computer vision tasks such as image understanding and generation from language descriptions. Yet, their potential for low-level tasks such as image restoration remains relatively unexplored. In this paper, we explore such models to enhance image restoration. As off-the-shelf features (OSF) from pre-trained models do not directly serve image restoration, we propose to learn an additional lightweight module called Pre-Train-Guided Refinement Module (PTG-RM) to refine restoration results of a target restoration network with OSF. PTG-RM consists of two components, Pre-Train-Guided Spatial-Varying Enhancement (PTG-SVE), and Pre-Train-Guided Channel-Spatial Attention (PTG-CSA). PTG-SVE enables optimal short- and long-range neural operations, while PTG-CSA enhances spatial-channel attention for restoration-related learning. Extensive experiments demonstrate that PTG-RM, with its compact size ($<$1M parameters), effectively enhances restoration performance of various models across different tasks, including low-light enhancement, deraining, deblurring, and denoising.

Related papers

• GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction [36.23508672036131]
  We propose an instance-Guided Low-rank Multi-Head selfattention to replace the Channel-wise Self-Attention. Unique to the proposed GLMHA is its ability to provide computational gain for both short and long input sequences. Our results show up to a 7.7 Giga FLOPs reduction with 370K fewer parameters required to closely retain the original performance of the best-performing models.
  arXiv  Detail & Related papers  (2024-10-01T04:07:48Z)
• Review Learning: Advancing All-in-One Ultra-High-Definition Image Restoration Training Method [7.487270862599671]
  We propose a new training paradigm for general image restoration models, which we name bfReview Learning. This approach begins with sequential training of an image restoration model on several degraded datasets, combined with a review mechanism. We design a lightweight all-purpose image restoration network that can efficiently reason about degraded images with 4K resolution on a single consumer-grade GPU.
  arXiv  Detail & Related papers  (2024-08-13T08:08:45Z)
• Fast Unsupervised Tensor Restoration via Low-rank Deconvolution [0.09208007322096533]
  Low-rank Deconvolution (LRD) has appeared as a new multi-dimensional representation model that enjoys important efficiency and flexibility properties. We ask ourselves if this analytical model can compete against Deep Learning (DL) frameworks like Deep Image Prior (DIP) or Blind-Spot Networks (BSN)
  arXiv  Detail & Related papers  (2024-06-15T16:04:49Z)
• AdaIR: Exploiting Underlying Similarities of Image Restoration Tasks with Adapters [57.62742271140852]
  AdaIR is a novel framework that enables low storage cost and efficient training without sacrificing performance. AdaIR requires solely the training of lightweight, task-specific modules, ensuring a more efficient storage and training regimen.
  arXiv  Detail & Related papers  (2024-04-17T15:31:06Z)
• Lightweight Adaptive Feature De-drifting for Compressed Image Classification [10.265991649449507]
  High-level vision models trained on high-quality images will suffer performance degradation when dealing with compressed images. Various learning-based JPEG artifact removal methods have been proposed to handle visual artifacts. This paper proposes a novel lightweight AFD module to boost the performance of pre-trained image classification models when facing compressed images.
  arXiv  Detail & Related papers  (2024-01-03T13:03:44Z)
• Multi-task Image Restoration Guided By Robust DINO Features [88.74005987908443]
  We propose mboxtextbfDINO-IR, a multi-task image restoration approach leveraging robust features extracted from DINOv2. We first propose a pixel-semantic fusion (PSF) module to dynamically fuse DINOV2's shallow features. By formulating these modules into a unified deep model, we propose a DINO perception contrastive loss to constrain the model training.
  arXiv  Detail & Related papers  (2023-12-04T06:59:55Z)
• HAT: Hybrid Attention Transformer for Image Restoration [61.74223315807691]
  Transformer-based methods have shown impressive performance in image restoration tasks, such as image super-resolution and denoising. We propose a new Hybrid Attention Transformer (HAT) to activate more input pixels for better restoration. Our HAT achieves state-of-the-art performance both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-09-11T05:17:55Z)
• REPNP: Plug-and-Play with Deep Reinforcement Learning Prior for Robust Image Restoration [30.966005373669027]
  We propose a novel deep reinforcement learning (DRL) based framework dubbed RePNP. Results demonstrate that the proposed RePNP is robust to the observation model used in the. scheme dubbed RePNP. RePNP achieves better results subjective to model deviation with fewer model parameters.
  arXiv  Detail & Related papers  (2022-07-25T10:56:10Z)
• Pre-Trained Image Processing Transformer [95.93031793337613]
  We develop a new pre-trained model, namely, image processing transformer (IPT) We present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. IPT model is trained on these images with multi-heads and multi-tails.
  arXiv  Detail & Related papers  (2020-12-01T09:42:46Z)
• Structure-Preserving Super Resolution with Gradient Guidance [87.79271975960764]
  Structures matter in single image super resolution (SISR) Recent studies benefiting from generative adversarial network (GAN) have promoted the development of SISR. However, there are always undesired structural distortions in the recovered images.
  arXiv  Detail & Related papers  (2020-03-29T17:26:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.