Learning Frequency-Aware Dynamic Transformers for All-In-One Image Restoration

• URL: http://arxiv.org/abs/2407.01636v1
• Date: Sun, 30 Jun 2024 13:14:44 GMT
• Title: Learning Frequency-Aware Dynamic Transformers for All-In-One Image Restoration
• Authors: Zenglin Shi, Tong Su, Pei Liu, Yunpeng Wu, Le Zhang, Meng Wang,
• Abstract summary: We propose a new all-in-one image restoration approach from a frequency perspective, leveraging advanced vision transformers. Our method consists of two main components: a frequency-aware Degradation prior learning transformer (Dformer) and a degradation-adaptive Restoration transformer (Rformer)
• Score: 15.561713188178455
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This work aims to tackle the all-in-one image restoration task, which seeks to handle multiple types of degradation with a single model. The primary challenge is to extract degradation representations from the input degraded images and use them to guide the model's adaptation to specific degradation types. Recognizing that various degradations affect image content differently across frequency bands, we propose a new all-in-one image restoration approach from a frequency perspective, leveraging advanced vision transformers. Our method consists of two main components: a frequency-aware Degradation prior learning transformer (Dformer) and a degradation-adaptive Restoration transformer (Rformer). The Dformer captures the essential characteristics of various degradations by decomposing inputs into different frequency components. By understanding how degradations affect these frequency components, the Dformer learns robust priors that effectively guide the restoration process. The Rformer then employs a degradation-adaptive self-attention module to selectively focus on the most affected frequency components, guided by the learned degradation representations. Extensive experimental results demonstrate that our approach outperforms the existing methods on four representative restoration tasks, including denoising, deraining, dehazing and deblurring. Additionally, our method offers benefits for handling spatially variant degradations and unseen degradation levels.

Related papers

• Bracketing Image Restoration and Enhancement with High-Low Frequency Decomposition [44.80645807358389]
  HLNet is a Bracketing Image Restoration and Enhancement method based on high-low frequency decomposition. We employ two modules for feature extraction: shared weight modules and non-shared weight modules. In the non-shared weight modules, we introduce the High-Low Frequency Decomposition Block (HLFDB), which employs different methods to handle high-low frequency information.
  arXiv  Detail & Related papers  (2024-04-21T05:11:37Z)
• Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration [44.35766203309201]
  We develop a Frequency Prompting image restoration method, dubbed FPro, which can effectively provide prompt components from a frequency perspective to guild the restoration model. Experimental results on commonly used benchmarks have demonstrated the favorable performance of our pipeline against SOTA methods on 5 image restoration tasks.
  arXiv  Detail & Related papers  (2024-03-30T08:42:34Z)
• AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulation [99.57024606542416]
  We propose an adaptive all-in-one image restoration network based on frequency mining and modulation. Our approach is motivated by the observation that different degradation types impact the image content on different frequency subbands. The proposed model achieves adaptive reconstruction by accentuating the informative frequency subbands according to different input degradations.
  arXiv  Detail & Related papers  (2024-03-21T17:58:14Z)
• Decoupling Degradation and Content Processing for Adverse Weather Image Restoration [79.59228846484415]
  Adverse weather image restoration strives to recover clear images from those affected by various weather types, such as rain, haze, and snow. Previous techniques can handle multiple weather types within a single network, but they neglect the crucial distinction between these two processes, limiting the quality of restored images. This work introduces a novel adverse weather image restoration method, called DDCNet, which decouples the degradation removal and content reconstruction process at the feature level based on their channel statistics.
  arXiv  Detail & Related papers  (2023-12-08T12:26:38Z)
• All-in-one Multi-degradation Image Restoration Network via Hierarchical Degradation Representation [47.00239809958627]
  We propose a novel All-in-one Multi-degradation Image Restoration Network (AMIRNet) AMIRNet learns a degradation representation for unknown degraded images by progressively constructing a tree structure through clustering. This tree-structured representation explicitly reflects the consistency and discrepancy of various distortions, providing a specific clue for image restoration.
  arXiv  Detail & Related papers  (2023-08-06T04:51:41Z)
• PromptIR: Prompting for All-in-One Blind Image Restoration [64.02374293256001]
  We present a prompt-based learning approach, PromptIR, for All-In-One image restoration. Our method uses prompts to encode degradation-specific information, which is then used to dynamically guide the restoration network. PromptIR offers a generic and efficient plugin module with few lightweight prompts.
  arXiv  Detail & Related papers  (2023-06-22T17:59:52Z)
• Image Deblurring by Exploring In-depth Properties of Transformer [86.7039249037193]
  We leverage deep features extracted from a pretrained vision transformer (ViT) to encourage recovered images to be sharp without sacrificing the performance measured by the quantitative metrics. By comparing the transformer features between recovered image and target one, the pretrained transformer provides high-resolution blur-sensitive semantic information. One regards the features as vectors and computes the discrepancy between representations extracted from recovered image and target one in Euclidean space.
  arXiv  Detail & Related papers  (2023-03-24T14:14:25Z)
• DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration [66.01846902242355]
  Blind face restoration usually synthesizes degraded low-quality data with a pre-defined degradation model for training. It is expensive and infeasible to include every type of degradation to cover real-world cases in the training data. We propose Robust Degradation Remover (DR2) to first transform the degraded image to a coarse but degradation-invariant prediction, then employ an enhancement module to restore the coarse prediction to a high-quality image.
  arXiv  Detail & Related papers  (2023-03-13T06:05:18Z)
• Invertible Rescaling Network and Its Extensions [118.72015270085535]
  In this work, we propose a novel invertible framework to model the bidirectional degradation and restoration from a new perspective. We develop invertible models to generate valid degraded images and transform the distribution of lost contents. Then restoration is made tractable by applying the inverse transformation on the generated degraded image together with a randomly-drawn latent variable.
  arXiv  Detail & Related papers  (2022-10-09T06:58:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.