Efficient Concertormer for Image Deblurring and Beyond

• URL: http://arxiv.org/abs/2404.06135v3
• Date: Wed, 04 Dec 2024 02:48:02 GMT
• Title: Efficient Concertormer for Image Deblurring and Beyond
• Authors: Pin-Hung Kuo, Jinshan Pan, Shao-Yi Chien, Ming-Hsuan Yang,
• Abstract summary: We introduce a novel Concerto Self-Attention (CSA) mechanism designed for image deblurring. By retaining partial information in additional dimensions independent from the self-attention calculations, our method effectively captures global contextual representations with complexity linear to the image size. While our primary objective is single-image motion deblurring, extensive quantitative and qualitative evaluations demonstrate that our approach performs favorably against state-of-the-art methods in other tasks.
• Score: 87.07963453448328
• License:
• Abstract: The Transformer architecture has achieved remarkable success in natural language processing and high-level vision tasks over the past few years. However, the inherent complexity of self-attention is quadratic to the size of the image, leading to unaffordable computational costs for high-resolution vision tasks. In this paper, we introduce Concertormer, featuring a novel Concerto Self-Attention (CSA) mechanism designed for image deblurring. The proposed CSA divides self-attention into two distinct components: one emphasizes generally global and another concentrates on specifically local correspondence. By retaining partial information in additional dimensions independent from the self-attention calculations, our method effectively captures global contextual representations with complexity linear to the image size. To effectively leverage the additional dimensions, we present a Cross-Dimensional Communication module, which linearly combines attention maps and thus enhances expressiveness. Moreover, we amalgamate the two-staged Transformer design into a single stage using the proposed gated-dconv MLP architecture. While our primary objective is single-image motion deblurring, extensive quantitative and qualitative evaluations demonstrate that our approach performs favorably against the state-of-the-art methods in other tasks, such as deraining and deblurring with JPEG artifacts. The source codes and trained models will be made available to the public.

Related papers

• SpotActor: Training-Free Layout-Controlled Consistent Image Generation [43.2870588035256]
  We present a new formalization of dual energy guidance with optimization in a dual semantic-latent space. We propose a training-free pipeline, SpotActor, which features a layout-conditioned backward update stage and a consistent forward sampling stage. The results prove that SpotActor fulfills the expectations of this task and showcases the potential for practical applications.
  arXiv  Detail & Related papers  (2024-09-07T11:52:48Z)
• ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling [11.129453244307369]
  FG-SBIR aims to minimize the distance between sketches and corresponding images in the embedding space. We propose an effective approach to narrow the gap between the two domains. It mainly facilitates unified mutual information sharing both intra- and inter-samples.
  arXiv  Detail & Related papers  (2024-06-17T13:49:12Z)
• Dual-Image Enhanced CLIP for Zero-Shot Anomaly Detection [58.228940066769596]
  We introduce a Dual-Image Enhanced CLIP approach, leveraging a joint vision-language scoring system. Our methods process pairs of images, utilizing each as a visual reference for the other, thereby enriching the inference process with visual context. Our approach significantly exploits the potential of vision-language joint anomaly detection and demonstrates comparable performance with current SOTA methods across various datasets.
  arXiv  Detail & Related papers  (2024-05-08T03:13:20Z)
• IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions [26.09373405194564]
  We present an efficient image processing transformer architecture with hierarchical attentions, called IPTV2. We adopt a focal context self-attention (FCSA) and a global grid self-attention (GGSA) to obtain adequate token interactions in local and global receptive fields. Our proposed IPT-V2 achieves state-of-the-art results on various image processing tasks, covering denoising, deblurring, deraining and obtains much better trade-off for performance and computational complexity than previous methods.
  arXiv  Detail & Related papers  (2024-03-31T10:01:20Z)
• Cross-Image Attention for Zero-Shot Appearance Transfer [68.43651329067393]
  We introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. We harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint.
  arXiv  Detail & Related papers  (2023-11-06T18:33:24Z)
• DARTS: Double Attention Reference-based Transformer for Super-resolution [12.424350934766704]
  We present DARTS, a transformer model for reference-based image super-resolution. DARS learns joint representations of two image distributions to enhance the content of low-resolution input images. We show that our transformer-based model performs competitively with state-of-the-art models.
  arXiv  Detail & Related papers  (2023-07-17T20:57:16Z)
• Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond [50.556961575275345]
  We build an image fusion module to fuse complementary characteristics and cascade dual task-related modules. We develop an efficient first-order approximation to compute corresponding gradients and present dynamic weighted aggregation to balance the gradients for fusion learning.
  arXiv  Detail & Related papers  (2023-05-11T10:55:34Z)
• TcGAN: Semantic-Aware and Structure-Preserved GANs with Individual Vision Transformer for Fast Arbitrary One-Shot Image Generation [11.207512995742999]
  One-shot image generation (OSG) with generative adversarial networks that learn from the internal patches of a given image has attracted world wide attention. We propose a novel structure-preserved method TcGAN with individual vision transformer to overcome the shortcomings of the existing one-shot image generation methods.
  arXiv  Detail & Related papers  (2023-02-16T03:05:59Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.