ATD: Improved Transformer with Adaptive Token Dictionary for Image Restoration

• URL: http://arxiv.org/abs/2603.02581v1
• Date: Tue, 03 Mar 2026 03:56:09 GMT
• Title: ATD: Improved Transformer with Adaptive Token Dictionary for Image Restoration
• Authors: Leheng Zhang, Wei Long, Yawei Li, Xingyu Zhou, Xiaorui Zhao, Shuhang Gu,
• Abstract summary: We propose Adaptive Token Dictionary (ATD), a novel transformer-based architecture for image restoration.<n>We exploit the category information embedded in the TDCA attention maps to group input features into multiple categories.<n>ATD and its lightweight version ATD-light, achieve state-of-the-art performance on multiple image super-resolution benchmarks.
• Score: 27.622615148357994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, Transformers have gained significant popularity in image restoration tasks such as image super-resolution and denoising, owing to their superior performance. However, balancing performance and computational burden remains a long-standing problem for transformer-based architectures. Due to the quadratic complexity of self-attention, existing methods often restrict attention to local windows, resulting in limited receptive field and suboptimal performance. To address this issue, we propose Adaptive Token Dictionary (ATD), a novel transformer-based architecture for image restoration that enables global dependency modeling with linear complexity relative to image size. The ATD model incorporates a learnable token dictionary, which summarizes external image priors (i.e., typical image structures) during the training process. To utilize this information, we introduce a token dictionary cross-attention (TDCA) mechanism that enhances the input features via interaction with the learned dictionary. Furthermore, we exploit the category information embedded in the TDCA attention maps to group input features into multiple categories, each representing a cluster of similar features across the image and serving as an attention group. We also integrate the learned category information into the feed-forward network to further improve feature fusion. ATD and its lightweight version ATD-light, achieve state-of-the-art performance on multiple image super-resolution benchmarks. Moreover, we develop ATD-U, a multi-scale variant of ATD, to address other image restoration tasks, including image denoising and JPEG compression artifacts removal. Extensive experiments demonstrate the superiority of out proposed models, both quantitatively and qualitatively.

Related papers

• Joint multi-dimensional dynamic attention and transformer for general image restoration [14.987034136856463]
  outdoor images often suffer from severe degradation due to rain, haze, and noise.<n>Current image restoration methods struggle to handle complex degradation while maintaining efficiency.<n>This paper introduces a novel image restoration architecture that combines multi-dimensional dynamic attention and self-attention.
  arXiv  Detail & Related papers  (2024-11-12T15:58:09Z)
• Multi-Scale Representation Learning for Image Restoration with State-Space Model [13.622411683295686]
  We propose a novel Multi-Scale State-Space Model-based (MS-Mamba) for efficient image restoration. Our proposed method achieves new state-of-the-art performance while maintaining low computational complexity.
  arXiv  Detail & Related papers  (2024-08-19T16:42:58Z)
• Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary [30.506135273928596]
  Single Image Super-Resolution is a classic computer vision problem that involves estimating high-resolution (HR) images from low-resolution (LR) ones. We introduce a group of auxiliary Adaptive Token Dictionary to SR Transformer and establish an ATD-SR method. Our method achieves the best performance on various single image super-resolution benchmarks.
  arXiv  Detail & Related papers  (2024-01-16T08:50:44Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• HAT: Hybrid Attention Transformer for Image Restoration [59.69448362233234]
  Transformer-based methods have shown impressive performance in image restoration tasks, such as image super-resolution and denoising.<n>We propose a new Hybrid Attention Transformer (HAT) to activate more input pixels for better restoration.<n>Our HAT achieves state-of-the-art performance both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-09-11T05:17:55Z)
• Prompt-based Ingredient-Oriented All-in-One Image Restoration [0.0]
  We propose a novel data ingredient-oriented approach to tackle multiple image degradation tasks. Specifically, we utilize a encoder to capture features and introduce prompts with degradation-specific information to guide the decoder. Our method performs competitively to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-09-06T15:05:04Z)
• DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation [56.514462874501675]
  We propose a dynamic sparse attention based Transformer model to achieve fine-level matching with favorable efficiency. The heart of our approach is a novel dynamic-attention unit, dedicated to covering the variation on the optimal number of tokens one position should focus on. Experiments on three applications, pose-guided person image generation, edge-based face synthesis, and undistorted image style transfer, demonstrate that DynaST achieves superior performance in local details.
  arXiv  Detail & Related papers  (2022-07-13T11:12:03Z)
• MAT: Mask-Aware Transformer for Large Hole Image Inpainting [79.67039090195527]
  We present a novel model for large hole inpainting, which unifies the merits of transformers and convolutions. Experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-29T06:36:17Z)
• XCiT: Cross-Covariance Image Transformers [73.33400159139708]
  We propose a "transposed" version of self-attention that operates across feature channels rather than tokens. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images.
  arXiv  Detail & Related papers  (2021-06-17T17:33:35Z)
• Less is More: Pay Less Attention in Vision Transformers [61.05787583247392]
  Less attention vIsion Transformer builds upon the fact that convolutions, fully-connected layers, and self-attentions have almost equivalent mathematical expressions for processing image patch sequences. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation.
  arXiv  Detail & Related papers  (2021-05-29T05:26:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.