HiTSR: A Hierarchical Transformer for Reference-based Super-Resolution

• URL: http://arxiv.org/abs/2408.16959v1
• Date: Fri, 30 Aug 2024 01:16:29 GMT
• Title: HiTSR: A Hierarchical Transformer for Reference-based Super-Resolution
• Authors: Masoomeh Aslahishahri, Jordan Ubbens, Ian Stavness,
• Abstract summary: We propose HiTSR, a hierarchical transformer model for reference-based image super-resolution. We streamline the architecture and training pipeline by incorporating the double attention block from GAN literature. Our model demonstrates superior performance across three datasets including SUN80, Urban100, and Manga109.
• Score: 6.546896650921257
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose HiTSR, a hierarchical transformer model for reference-based image super-resolution, which enhances low-resolution input images by learning matching correspondences from high-resolution reference images. Diverging from existing multi-network, multi-stage approaches, we streamline the architecture and training pipeline by incorporating the double attention block from GAN literature. Processing two visual streams independently, we fuse self-attention and cross-attention blocks through a gating attention strategy. The model integrates a squeeze-and-excitation module to capture global context from the input images, facilitating long-range spatial interactions within window-based attention blocks. Long skip connections between shallow and deep layers further enhance information flow. Our model demonstrates superior performance across three datasets including SUN80, Urban100, and Manga109. Specifically, on the SUN80 dataset, our model achieves PSNR/SSIM values of 30.24/0.821. These results underscore the effectiveness of attention mechanisms in reference-based image super-resolution. The transformer-based model attains state-of-the-art results without the need for purpose-built subnetworks, knowledge distillation, or multi-stage training, emphasizing the potency of attention in meeting reference-based image super-resolution requirements.

Related papers

• ECAFormer: Low-light Image Enhancement using Cross Attention [11.554554006307836]
  Low-light image enhancement (LLIE) is critical in computer vision. We design a hierarchical mutual Enhancement via a Cross Attention transformer (ECAFormer) We show that ECAFormer reaches competitive performance across multiple benchmarks, yielding nearly a 3% improvement in PSNR over the suboptimal method.
  arXiv  Detail & Related papers  (2024-06-19T07:21:31Z)
• IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions [26.09373405194564]
  We present an efficient image processing transformer architecture with hierarchical attentions, called IPTV2. We adopt a focal context self-attention (FCSA) and a global grid self-attention (GGSA) to obtain adequate token interactions in local and global receptive fields. Our proposed IPT-V2 achieves state-of-the-art results on various image processing tasks, covering denoising, deblurring, deraining and obtains much better trade-off for performance and computational complexity than previous methods.
  arXiv  Detail & Related papers  (2024-03-31T10:01:20Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• DARTS: Double Attention Reference-based Transformer for Super-resolution [12.424350934766704]
  We present DARTS, a transformer model for reference-based image super-resolution. DARS learns joint representations of two image distributions to enhance the content of low-resolution input images. We show that our transformer-based model performs competitively with state-of-the-art models.
  arXiv  Detail & Related papers  (2023-07-17T20:57:16Z)
• Cross-Spatial Pixel Integration and Cross-Stage Feature Fusion Based Transformer Network for Remote Sensing Image Super-Resolution [13.894645293832044]
  Transformer-based models have shown competitive performance in remote sensing image super-resolution (RSISR) We propose a novel transformer architecture called Cross-Spatial Pixel Integration and Cross-Stage Feature Fusion Based Transformer Network (SPIFFNet) for RSISR. Our proposed model effectively enhances global cognition and understanding of the entire image, facilitating efficient integration of features cross-stages.
  arXiv  Detail & Related papers  (2023-07-06T13:19:06Z)
• Cross-View Hierarchy Network for Stereo Image Super-Resolution [14.574538513341277]
  Stereo image super-resolution aims to improve the quality of high-resolution stereo image pairs by exploiting complementary information across views. We propose a novel method, named Cross-View-Hierarchy Network for Stereo Image Super-Resolution (CVHSSR) CVHSSR achieves the best stereo image super-resolution performance than other state-of-the-art methods while using fewer parameters.
  arXiv  Detail & Related papers  (2023-04-13T03:11:30Z)
• StraIT: Non-autoregressive Generation with Stratified Image Transformer [63.158996766036736]
  Stratified Image Transformer(StraIT) is a pure non-autoregressive(NAR) generative model. Our experiments demonstrate that StraIT significantly improves NAR generation and out-performs existing DMs and AR methods.
  arXiv  Detail & Related papers  (2023-03-01T18:59:33Z)
• Learning Enriched Features for Fast Image Restoration and Enhancement [166.17296369600774]
  This paper presents a holistic goal of maintaining spatially-precise high-resolution representations through the entire network. We learn an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Our approach achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement.
  arXiv  Detail & Related papers  (2022-04-19T17:59:45Z)
• High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion [84.24973877109181]
  We propose a novel attention-based hierarchical multi-modal fusion network for guided DSR. We show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.
  arXiv  Detail & Related papers  (2021-04-04T03:28:33Z)
• Normalizing Flows with Multi-Scale Autoregressive Priors [131.895570212956]
  We introduce channel-wise dependencies in their latent space through multi-scale autoregressive priors (mAR) Our mAR prior for models with split coupling flow layers (mAR-SCF) can better capture dependencies in complex multimodal data. We show that mAR-SCF allows for improved image generation quality, with gains in FID and Inception scores compared to state-of-the-art flow-based models.
  arXiv  Detail & Related papers  (2020-04-08T09:07:11Z)
• Learning Enriched Features for Real Image Restoration and Enhancement [166.17296369600774]
  convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. We present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network. Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.
  arXiv  Detail & Related papers  (2020-03-15T11:04:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.