UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior

• URL: http://arxiv.org/abs/2501.13134v1
• Date: Wed, 22 Jan 2025 08:06:48 GMT
• Title: UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior
• Authors: I-Hsiang Chen, Wei-Ting Chen, Yu-Wei Liu, Yuan-Chun Chiang, Sy-Yen Kuo, Ming-Hsuan Yang,
• Abstract summary: Image restoration aims to recover content from inputs degraded by various factors, such as adverse weather, blur, and noise. This paper introduces UniRestore, a unified image restoration model that bridges the gap between PIR and TIR. We propose a Complementary Feature Restoration Module (CFRM) to reconstruct degraded encoder features and a Task Feature Adapter (TFA) module to facilitate adaptive feature fusion in the decoder.
• Score: 56.35236964617809
• License:
• Abstract: Image restoration aims to recover content from inputs degraded by various factors, such as adverse weather, blur, and noise. Perceptual Image Restoration (PIR) methods improve visual quality but often do not support downstream tasks effectively. On the other hand, Task-oriented Image Restoration (TIR) methods focus on enhancing image utility for high-level vision tasks, sometimes compromising visual quality. This paper introduces UniRestore, a unified image restoration model that bridges the gap between PIR and TIR by using a diffusion prior. The diffusion prior is designed to generate images that align with human visual quality preferences, but these images are often unsuitable for TIR scenarios. To solve this limitation, UniRestore utilizes encoder features from an autoencoder to adapt the diffusion prior to specific tasks. We propose a Complementary Feature Restoration Module (CFRM) to reconstruct degraded encoder features and a Task Feature Adapter (TFA) module to facilitate adaptive feature fusion in the decoder. This design allows UniRestore to optimize images for both human perception and downstream task requirements, addressing discrepancies between visual quality and functional needs. Integrating these modules also enhances UniRestore's adapability and efficiency across diverse tasks. Extensive expertments demonstrate the superior performance of UniRestore in both PIR and TIR scenarios.

Related papers

• MoE-DiffIR: Task-customized Diffusion Priors for Universal Compressed Image Restoration [16.482022642533448]
  MoE-DiffIR is an innovative universal compressed image restoration (CIR) method with task-customized diffusion priors. MoE-DiffIR develops the powerful mixture-of-experts (MoE) prompt module. The degradation-aware routing mechanism is proposed to enable the flexible assignment of basic prompts.
  arXiv  Detail & Related papers  (2024-07-15T15:43:27Z)
• Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration [19.87693298262894]
  We propose Diff-Restorer, a universal image restoration method based on the diffusion model. We utilize the pre-trained visual language model to extract visual prompts from degraded images. We also design a Degradation-aware Decoder to perform structural correction and convert the latent code to the pixel domain.
  arXiv  Detail & Related papers  (2024-07-04T05:01:10Z)
• LIR: A Lightweight Baseline for Image Restoration [4.187190284830909]
  The inherent characteristics of the Image Restoration task are often overlooked in many works. We propose a Lightweight Baseline network for Image Restoration called LIR to efficiently restore the image and remove degradations. Our LIR achieves the state-of-the-art Structure Similarity Index Measure (SSIM) and comparable performance to state-of-the-art models on Peak Signal-to-Noise Ratio (PSNR)
  arXiv  Detail & Related papers  (2024-02-02T12:39:47Z)
• Unified-Width Adaptive Dynamic Network for All-In-One Image Restoration [50.81374327480445]
  We introduce a novel concept positing that intricate image degradation can be represented in terms of elementary degradation. We propose the Unified-Width Adaptive Dynamic Network (U-WADN), consisting of two pivotal components: a Width Adaptive Backbone (WAB) and a Width Selector (WS) The proposed U-WADN achieves better performance while simultaneously reducing up to 32.3% of FLOPs and providing approximately 15.7% real-time acceleration.
  arXiv  Detail & Related papers  (2024-01-24T04:25:12Z)
• Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration [58.11518043688793]
  MPerceiver is a novel approach to enhance adaptiveness, generalizability and fidelity for all-in-one image restoration. MPerceiver is trained on 9 tasks for all-in-one IR and outperforms state-of-the-art task-specific methods across most tasks.
  arXiv  Detail & Related papers  (2023-12-05T17:47:11Z)
• Multi-task Image Restoration Guided By Robust DINO Features [88.74005987908443]
  We propose mboxtextbfDINO-IR, a multi-task image restoration approach leveraging robust features extracted from DINOv2. We first propose a pixel-semantic fusion (PSF) module to dynamically fuse DINOV2's shallow features. By formulating these modules into a unified deep model, we propose a DINO perception contrastive loss to constrain the model training.
  arXiv  Detail & Related papers  (2023-12-04T06:59:55Z)
• Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model [59.08821399652483]
  Illumination degradation image restoration (IDIR) techniques aim to improve the visibility of degraded images and mitigate the adverse effects of deteriorated illumination. Among these algorithms, diffusion model (DM)-based methods have shown promising performance but are often burdened by heavy computational demands and pixel misalignment issues when predicting the image-level distribution. We propose to leverage DM within a compact latent space to generate concise guidance priors and introduce a novel solution called Reti-Diff for the IDIR task. Reti-Diff comprises two key components: the Retinex-based latent DM (RLDM) and the Retinex-guided transformer (RG
  arXiv  Detail & Related papers  (2023-11-20T09:55:06Z)
• DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior [70.46245698746874]
  We present DiffBIR, a general restoration pipeline that could handle different blind image restoration tasks. DiffBIR decouples blind image restoration problem into two stages: 1) degradation removal: removing image-independent content; 2) information regeneration: generating the lost image content. In the first stage, we use restoration modules to remove degradations and obtain high-fidelity restored results. For the second stage, we propose IRControlNet that leverages the generative ability of latent diffusion models to generate realistic details.
  arXiv  Detail & Related papers  (2023-08-29T07:11:52Z)
• DRM-IR: Task-Adaptive Deep Unfolding Network for All-In-One Image Restoration [5.573836220587265]
  This work proposes an efficient Dynamic Reference Modeling paradigm (DRM-IR) DRM-IR consists of task-adaptive degradation modeling and model-based image restoring. Experiments on multiple benchmark datasets show that our DRM-IR achieves state-of-the-art in All-In-One IR.
  arXiv  Detail & Related papers  (2023-07-15T02:42:19Z)
• A Dive into SAM Prior in Image Restoration [40.03648504115027]
  The goal of image restoration (IR) is to restore a high-quality (HQ) image from its degraded low-quality (LQ) observation. We propose a lightweight SAM prior tuning (SPT) unit to integrate semantic priors into existing IR networks. As the only trainable module in our method, the SPT unit has the potential to improve both efficiency and scalability.
  arXiv  Detail & Related papers  (2023-05-23T02:31:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.