Joint multi-dimensional dynamic attention and transformer for general image restoration

• URL: http://arxiv.org/abs/2411.07893v1
• Date: Tue, 12 Nov 2024 15:58:09 GMT
• Title: Joint multi-dimensional dynamic attention and transformer for general image restoration
• Authors: Huan Zhang, Xu Zhang, Nian Cai, Jianglei Di, Yun Zhang,
• Abstract summary: outdoor images often suffer from severe degradation due to rain, haze, and noise. Current image restoration methods struggle to handle complex degradation while maintaining efficiency. This paper introduces a novel image restoration architecture that combines multi-dimensional dynamic attention and self-attention.
• Score: 14.987034136856463
• License:
• Abstract: Outdoor images often suffer from severe degradation due to rain, haze, and noise, impairing image quality and challenging high-level tasks. Current image restoration methods struggle to handle complex degradation while maintaining efficiency. This paper introduces a novel image restoration architecture that combines multi-dimensional dynamic attention and self-attention within a U-Net framework. To leverage the global modeling capabilities of transformers and the local modeling capabilities of convolutions, we integrate sole CNNs in the encoder-decoder and sole transformers in the latent layer. Additionally, we design convolutional kernels with selected multi-dimensional dynamic attention to capture diverse degraded inputs efficiently. A transformer block with transposed self-attention further enhances global feature extraction while maintaining efficiency. Extensive experiments demonstrate that our method achieves a better balance between performance and computational complexity across five image restoration tasks: deraining, deblurring, denoising, dehazing, and enhancement, as well as superior performance for high-level vision tasks. The source code will be available at https://github.com/House-yuyu/MDDA-former.

Related papers

• Multi-Scale Representation Learning for Image Restoration with State-Space Model [13.622411683295686]
  We propose a novel Multi-Scale State-Space Model-based (MS-Mamba) for efficient image restoration. Our proposed method achieves new state-of-the-art performance while maintaining low computational complexity.
  arXiv  Detail & Related papers  (2024-08-19T16:42:58Z)
• VmambaIR: Visual State Space Model for Image Restoration [36.11385876754612]
  We propose VmambaIR, which introduces State Space Models (SSMs) with linear complexity into comprehensive image restoration tasks. VmambaIR achieves state-of-the-art (SOTA) performance with much fewer computational resources and parameters.
  arXiv  Detail & Related papers  (2024-03-18T02:38:55Z)
• Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration [91.65248635837145]
  Under-Display Camera (UDC) is an emerging technology that achieves full-screen display via hiding the camera under the display panel. In this paper, we observe that when using the Vision Transformer for UDC degraded image restoration, the global attention samples a large amount of redundant information and noise. We propose a Guided Sparse Transformer method (SGSFormer) for the task of restoring high-quality images from UDC degraded images.
  arXiv  Detail & Related papers  (2024-03-09T13:11:59Z)
• CascadedGaze: Efficiency in Global Context Extraction for Image Restoration [12.967835674413596]
  We present CascadedGaze Network (CGNet), an encoder-decoder architecture that employs Global Context Extractor (GCE) The GCE module leverages small kernels across convolutional layers to learn global dependencies, without requiring self-attention.
  arXiv  Detail & Related papers  (2024-01-26T22:59:51Z)
• HAT: Hybrid Attention Transformer for Image Restoration [61.74223315807691]
  Transformer-based methods have shown impressive performance in image restoration tasks, such as image super-resolution and denoising. We propose a new Hybrid Attention Transformer (HAT) to activate more input pixels for better restoration. Our HAT achieves state-of-the-art performance both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-09-11T05:17:55Z)
• Efficient Image Super-Resolution with Feature Interaction Weighted Hybrid Network [101.53907377000445]
  Lightweight image super-resolution aims to reconstruct high-resolution images from low-resolution images using low computational costs. Existing methods result in the loss of middle-layer features due to activation functions. We propose a Feature Interaction Weighted Hybrid Network (FIWHN) to minimize the impact of intermediate feature loss on reconstruction quality.
  arXiv  Detail & Related papers  (2022-12-29T05:57:29Z)
• DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation [56.514462874501675]
  We propose a dynamic sparse attention based Transformer model to achieve fine-level matching with favorable efficiency. The heart of our approach is a novel dynamic-attention unit, dedicated to covering the variation on the optimal number of tokens one position should focus on. Experiments on three applications, pose-guided person image generation, edge-based face synthesis, and undistorted image style transfer, demonstrate that DynaST achieves superior performance in local details.
  arXiv  Detail & Related papers  (2022-07-13T11:12:03Z)
• MAT: Mask-Aware Transformer for Large Hole Image Inpainting [79.67039090195527]
  We present a novel model for large hole inpainting, which unifies the merits of transformers and convolutions. Experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-29T06:36:17Z)
• Restormer: Efficient Transformer for High-Resolution Image Restoration [118.9617735769827]
  convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data. Transformers have shown significant performance gains on natural language and high-level vision tasks. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks.
  arXiv  Detail & Related papers  (2021-11-18T18:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.