HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Models

• URL: http://arxiv.org/abs/2311.17528v2
• Date: Mon, 29 Apr 2024 09:26:36 GMT
• Title: HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Models
• Authors: Shen Zhang, Zhaowei Chen, Zhenyu Zhao, Yuhao Chen, Yao Tang, Jiajun Liang,
• Abstract summary: HiDiffusion is a tuning-free higher-resolution framework for image synthesis. RAU-Net dynamically adjusts the feature map size to resolve object duplication. MSW-MSA engages optimized window attention to reduce computations.
• Score: 13.68666823175341
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have become a mainstream approach for high-resolution image synthesis. However, directly generating higher-resolution images from pretrained diffusion models will encounter unreasonable object duplication and exponentially increase the generation time. In this paper, we discover that object duplication arises from feature duplication in the deep blocks of the U-Net. Concurrently, We pinpoint the extended generation times to self-attention redundancy in U-Net's top blocks. To address these issues, we propose a tuning-free higher-resolution framework named HiDiffusion. Specifically, HiDiffusion contains Resolution-Aware U-Net (RAU-Net) that dynamically adjusts the feature map size to resolve object duplication and engages Modified Shifted Window Multi-head Self-Attention (MSW-MSA) that utilizes optimized window attention to reduce computations. we can integrate HiDiffusion into various pretrained diffusion models to scale image generation resolutions even to 4096x4096 at 1.5-6x the inference speed of previous methods. Extensive experiments demonstrate that our approach can address object duplication and heavy computation issues, achieving state-of-the-art performance on higher-resolution image synthesis tasks.

Related papers

• High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity [69.32473738284374]
  We propose DiffDIS, a diffusion-driven segmentation model that taps into the potential of the pre-trained U-Net within diffusion models. By leveraging the robust generalization capabilities and rich, versatile image representation prior to the SD models, we significantly reduce the inference time while preserving high-fidelity, detailed generation. Experiments on the DIS5K dataset demonstrate the superiority of DiffDIS, achieving state-of-the-art results through a streamlined inference process.
  arXiv  Detail & Related papers  (2024-10-14T02:49:23Z)
• DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidance [11.44012694656102]
  Large-scale generative models, such as text-to-image diffusion models, have garnered widespread attention across diverse domains. Existing large-scale diffusion models are confined to generating images of up to 1K resolution. We propose a novel progressive approach that fully utilizes generated low-resolution images to guide the generation of higher-resolution images.
  arXiv  Detail & Related papers  (2024-06-26T16:10:31Z)
• Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models [26.926712014346432]
  This paper presents innovative enhancements to diffusion models by integrating a novel multi-resolution network and time-dependent layer normalization. Our method's efficacy is demonstrated on the class-conditional ImageNet generation benchmark, setting new state-of-the-art FID scores of 1.70 on ImageNet 256 x 256 and 2.89 on ImageNet 512 x 512.
  arXiv  Detail & Related papers  (2024-06-13T17:59:58Z)
• Binarized Diffusion Model for Image Super-Resolution [61.963833405167875]
  Binarization, an ultra-compression algorithm, offers the potential for effectively accelerating advanced diffusion models (DMs) Existing binarization methods result in significant performance degradation. We introduce a novel binarized diffusion model, BI-DiffSR, for image SR.
  arXiv  Detail & Related papers  (2024-06-09T10:30:25Z)
• ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models [126.35334860896373]
  We investigate the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes. Existing works for higher-resolution generation, such as attention-based and joint-diffusion approaches, cannot well address these issues. We propose a simple yet effective re-dilation that can dynamically adjust the convolutional perception field during inference.
  arXiv  Detail & Related papers  (2023-10-11T17:52:39Z)
• Hierarchical Integration Diffusion Model for Realistic Image Deblurring [71.76410266003917]
  Diffusion models (DMs) have been introduced in image deblurring and exhibited promising performance. We propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T12:18:20Z)
• Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Models [9.245782611878752]
  We enhance the diffusion model in several aspects such as network architecture, noise level, denoising steps, training image size, and perceptual/scheduler scores. We also propose a U-Net based latent diffusion model which performs diffusion in a low-resolution latent space while preserving high-resolution information from the original input for the decoding process. These modifications allow us to apply diffusion models to various image restoration tasks, including real-world shadow removal, HR non-homogeneous dehazing, stereo super-resolution, and bokeh effect transformation.
  arXiv  Detail & Related papers  (2023-04-17T14:06:49Z)
• Implicit Diffusion Models for Continuous Super-Resolution [65.45848137914592]
  This paper introduces an Implicit Diffusion Model (IDM) for high-fidelity continuous image super-resolution. IDM integrates an implicit neural representation and a denoising diffusion model in a unified end-to-end framework. The scaling factor regulates the resolution and accordingly modulates the proportion of the LR information and generated features in the final output.
  arXiv  Detail & Related papers  (2023-03-29T07:02:20Z)
• High-Resolution Image Editing via Multi-Stage Blended Diffusion [3.834509400202395]
  We propose an approach that uses a pre-trained low-resolution diffusion model to edit images in the megapixel range. We first use Blended Diffusion to edit the image at a low resolution, and then upscale it in multiple stages, using a super-resolution model and Blended Diffusion.
  arXiv  Detail & Related papers  (2022-10-24T06:07:35Z)
• Dynamic Dual-Output Diffusion Models [100.32273175423146]
  Iterative denoising-based generation has been shown to be comparable in quality to other classes of generative models. A major drawback of this method is that it requires hundreds of iterations to produce a competitive result. Recent works have proposed solutions that allow for faster generation with fewer iterations, but the image quality gradually deteriorates.
  arXiv  Detail & Related papers  (2022-03-08T11:20:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.