PatchScaler: An Efficient Patch-Independent Diffusion Model for Image Super-Resolution

• URL: http://arxiv.org/abs/2405.17158v4
• Date: Thu, 21 Nov 2024 12:35:18 GMT
• Title: PatchScaler: An Efficient Patch-Independent Diffusion Model for Image Super-Resolution
• Authors: Yong Liu, Hang Dong, Jinshan Pan, Qingji Dong, Kai Chen, Rongxiang Zhang, Lean Fu, Fei Wang,
• Abstract summary: PatchScaler is an efficient patch-independent diffusion pipeline for single image super-resolution. A texture prompt adaptively retrieves texture priors for the target patch from a common reference texture memory. Our code achieves superior performance in both quantitative and qualitative evaluations, while significantly speeding up inference.
• Score: 44.345740602726345
• License:
• Abstract: While diffusion models significantly improve the perceptual quality of super-resolved images, they usually require a large number of sampling steps, resulting in high computational costs and long inference times. Recent efforts have explored reasonable acceleration schemes by reducing the number of sampling steps. However, these approaches treat all regions of the image equally, overlooking the fact that regions with varying levels of reconstruction difficulty require different sampling steps. To address this limitation, we propose PatchScaler, an efficient patch-independent diffusion pipeline for single image super-resolution. Specifically, PatchScaler introduces a Patch-adaptive Group Sampling (PGS) strategy that groups feature patches by quantifying their reconstruction difficulty and establishes shortcut paths with different sampling configurations for each group. To further optimize the patch-level reconstruction process of PGS, we propose a texture prompt that provides rich texture conditional information to the diffusion model. The texture prompt adaptively retrieves texture priors for the target patch from a common reference texture memory. Extensive experiments show that our PatchScaler achieves superior performance in both quantitative and qualitative evaluations, while significantly speeding up inference. Our code will be available at \url{https://github.com/yongliuy/PatchScaler}.

Related papers

• PassionSR: Post-Training Quantization with Adaptive Scale in One-Step Diffusion based Image Super-Resolution [87.89013794655207]
  Diffusion-based image super-resolution (SR) models have shown superior performance at the cost of multiple denoising steps. We propose a novel post-training quantization approach with adaptive scale in one-step diffusion (OSD) image SR, PassionSR. Our PassionSR achieves significant advantages over recent leading low-bit quantization methods for image SR.
  arXiv  Detail & Related papers  (2024-11-26T04:49:42Z)
• EPS: Efficient Patch Sampling for Video Overfitting in Deep Super-Resolution Model Training [15.684865589513597]
  We propose an efficient patch sampling method named EPS for video SR network overfitting. Our method reduces the number of patches for the training to 4% to 25%, depending on the resolution and number of clusters. Compared to the state-of-the-art patch sampling method, EMT, our approach achieves an 83% decrease in overall run time.
  arXiv  Detail & Related papers  (2024-11-25T12:01:57Z)
• Adaptive Patching for High-resolution Image Segmentation with Transformers [9.525013089622183]
  Attention-based models are proliferating in the space of image analytics, including segmentation. Standard method of feeding images to transformer encoders is to divide the images into patches and then feed the patches to the model as a linear sequence of tokens. For high-resolution images, e.g. microscopic pathology images, the quadratic compute and memory cost prohibits the use of an attention-based model, if we are to use smaller patch sizes that are favorable in segmentation. We take inspiration from Adapative Mesh Refinement (AMR) methods in HPC by adaptively patching the images, as a pre-processing step, based
  arXiv  Detail & Related papers  (2024-04-15T12:06:00Z)
• EXTRACTER: Efficient Texture Matching with Attention and Gradient Enhancing for Large Scale Image Super Resolution [0.0]
  Recent Reference-Based image super-resolution (RefSR) has improved SOTA deep methods introducing attention mechanisms to enhance low-resolution images. We propose a deep search with a more efficient memory usage that reduces significantly the number of image patches.
  arXiv  Detail & Related papers  (2023-10-02T17:41:56Z)
• DBAT: Dynamic Backward Attention Transformer for Material Segmentation with Cross-Resolution Patches [8.812837829361923]
  We propose the Dynamic Backward Attention Transformer (DBAT) to aggregate cross-resolution features. Experiments show that our DBAT achieves an accuracy of 86.85%, which is the best performance among state-of-the-art real-time models. We further align features to semantic labels, performing network dissection, to infer that the proposed model can extract material-related features better than other methods.
  arXiv  Detail & Related papers  (2023-05-06T03:47:20Z)
• FewGAN: Generating from the Joint Distribution of a Few Images [95.6635227371479]
  We introduce FewGAN, a generative model for generating novel, high-quality and diverse images. FewGAN is a hierarchical patch-GAN that applies quantization at the first coarse scale, followed by a pyramid of residual fully convolutional GANs at finer scales. In an extensive set of experiments, it is shown that FewGAN outperforms baselines both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2022-07-18T07:11:28Z)
• HIPA: Hierarchical Patch Transformer for Single Image Super Resolution [62.7081074931892]
  This paper presents HIPA, a novel Transformer architecture that progressively recovers the high resolution image using a hierarchical patch partition. We build a cascaded model that processes an input image in multiple stages, where we start with tokens with small patch sizes and gradually merge to the full resolution. Such a hierarchical patch mechanism not only explicitly enables feature aggregation at multiple resolutions but also adaptively learns patch-aware features for different image regions.
  arXiv  Detail & Related papers  (2022-03-19T05:09:34Z)
• SDWNet: A Straight Dilated Network with Wavelet Transformation for Image Deblurring [23.86692375792203]
  Image deblurring is a computer vision problem that aims to recover a sharp image from a blurred image. Our model uses dilated convolution to enable the obtainment of the large receptive field with high spatial resolution. We propose a novel module using the wavelet transform, which effectively helps the network to recover clear high-frequency texture details.
  arXiv  Detail & Related papers  (2021-10-12T07:58:10Z)
• Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform [58.60004238261117]
  We propose a versatile deep image compression network based on Spatial Feature Transform (SFT arXiv:1804.02815) Our model covers a wide range of compression rates using a single model, which is controlled by arbitrary pixel-wise quality maps. The proposed framework allows us to perform task-aware image compressions for various tasks.
  arXiv  Detail & Related papers  (2021-08-21T17:30:06Z)
• A Hierarchical Transformation-Discriminating Generative Model for Few Shot Anomaly Detection [93.38607559281601]
  We devise a hierarchical generative model that captures the multi-scale patch distribution of each training image. The anomaly score is obtained by aggregating the patch-based votes of the correct transformation across scales and image regions.
  arXiv  Detail & Related papers  (2021-04-29T17:49:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.