A Dive into SAM Prior in Image Restoration

• URL: http://arxiv.org/abs/2305.13620v1
• Date: Tue, 23 May 2023 02:31:06 GMT
• Title: A Dive into SAM Prior in Image Restoration
• Authors: Zeyu Xiao, Jiawang Bai, Zhihe Lu, Zhiwei Xiong
• Abstract summary: The goal of image restoration (IR) is to restore a high-quality (HQ) image from its degraded low-quality (LQ) observation. We propose a lightweight SAM prior tuning (SPT) unit to integrate semantic priors into existing IR networks. As the only trainable module in our method, the SPT unit has the potential to improve both efficiency and scalability.
• Score: 40.03648504115027
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The goal of image restoration (IR), a fundamental issue in computer vision, is to restore a high-quality (HQ) image from its degraded low-quality (LQ) observation. Multiple HQ solutions may correspond to an LQ input in this poorly posed problem, creating an ambiguous solution space. This motivates the investigation and incorporation of prior knowledge in order to effectively constrain the solution space and enhance the quality of the restored images. In spite of the pervasive use of hand-crafted and learned priors in IR, limited attention has been paid to the incorporation of knowledge from large-scale foundation models. In this paper, we for the first time leverage the prior knowledge of the state-of-the-art segment anything model (SAM) to boost the performance of existing IR networks in an parameter-efficient tuning manner. In particular, the choice of SAM is based on its robustness to image degradations, such that HQ semantic masks can be extracted from it. In order to leverage semantic priors and enhance restoration quality, we propose a lightweight SAM prior tuning (SPT) unit. This plug-and-play component allows us to effectively integrate semantic priors into existing IR networks, resulting in significant improvements in restoration quality. As the only trainable module in our method, the SPT unit has the potential to improve both efficiency and scalability. We demonstrate the effectiveness of the proposed method in enhancing a variety of methods across multiple tasks, such as image super-resolution and color image denoising.

Related papers

• Haze-Aware Attention Network for Single-Image Dehazing [10.881567541939653]
  We propose a new dehazing network combining an innovative Haze-Aware Attention Module (HAAM) with a Multiscale Frequency Enhancement Module (MFEM) The HAAM is inspired by the atmospheric scattering model, thus skillfully integrating physical principles into high-dimensional features for targeted dehazing. Our work not only advances the field of image dehazing but also offers insights into the design of attention mechanisms for broader applications in computer vision.
  arXiv  Detail & Related papers  (2024-07-16T08:42:39Z)
• Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration [19.87693298262894]
  We propose Diff-Restorer, a universal image restoration method based on the diffusion model. We utilize the pre-trained visual language model to extract visual prompts from degraded images. We also design a Degradation-aware Decoder to perform structural correction and convert the latent code to the pixel domain.
  arXiv  Detail & Related papers  (2024-07-04T05:01:10Z)
• PriorNet: A Novel Lightweight Network with Multidimensional Interactive Attention for Efficient Image Dehazing [8.837086917206525]
  Hazy images degrade visual quality, and dehazing is a crucial prerequisite for subsequent processing tasks. This paper introduces PriorNet, a novel, lightweight, and highly applicable dehazing network. The core of PriorNet is the original Multi-Dimensional Interactive Attention (MIA) mechanism, which effectively captures a wide range of haze characteristics.
  arXiv  Detail & Related papers  (2024-04-24T04:20:22Z)
• LIR: A Lightweight Baseline for Image Restoration [4.187190284830909]
  The inherent characteristics of the Image Restoration task are often overlooked in many works. We propose a Lightweight Baseline network for Image Restoration called LIR to efficiently restore the image and remove degradations. Our LIR achieves the state-of-the-art Structure Similarity Index Measure (SSIM) and comparable performance to state-of-the-art models on Peak Signal-to-Noise Ratio (PSNR)
  arXiv  Detail & Related papers  (2024-02-02T12:39:47Z)
• Exploring Resolution and Degradation Clues as Self-supervised Signal for Low Quality Object Detection [77.3530907443279]
  We propose a novel self-supervised framework to detect objects in degraded low resolution images. Our methods has achieved superior performance compared with existing methods when facing variant degradation situations.
  arXiv  Detail & Related papers  (2022-08-05T09:36:13Z)
• Rank-Enhanced Low-Dimensional Convolution Set for Hyperspectral Image Denoising [50.039949798156826]
  This paper tackles the challenging problem of hyperspectral (HS) image denoising. We propose rank-enhanced low-dimensional convolution set (Re-ConvSet) We then incorporate Re-ConvSet into the widely-used U-Net architecture to construct an HS image denoising method.
  arXiv  Detail & Related papers  (2022-07-09T13:35:12Z)
• Hierarchical Similarity Learning for Aliasing Suppression Image Super-Resolution [64.15915577164894]
  A hierarchical image super-resolution network (HSRNet) is proposed to suppress the influence of aliasing. HSRNet achieves better quantitative and visual performance than other works, and remits the aliasing more effectively.
  arXiv  Detail & Related papers  (2022-06-07T14:55:32Z)
• Learning Enriched Features for Real Image Restoration and Enhancement [166.17296369600774]
  convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. We present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network. Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.
  arXiv  Detail & Related papers  (2020-03-15T11:04:30Z)
• Gated Fusion Network for Degraded Image Super Resolution [78.67168802945069]
  We propose a dual-branch convolutional neural network to extract base features and recovered features separately. By decomposing the feature extraction step into two task-independent streams, the dual-branch model can facilitate the training process.
  arXiv  Detail & Related papers  (2020-03-02T13:28:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.