HiPA: Enabling One-Step Text-to-Image Diffusion Models via High-Frequency-Promoting Adaptation

• URL: http://arxiv.org/abs/2311.18158v1
• Date: Thu, 30 Nov 2023 00:14:07 GMT
• Title: HiPA: Enabling One-Step Text-to-Image Diffusion Models via High-Frequency-Promoting Adaptation
• Authors: Yifan Zhang, Bryan Hooi
• Abstract summary: High-frequency-Promoting Adaptation (HiPA) is a parameter-efficient approach to enable one-step text-to-image diffusion. HiPA focuses on training one-step, low-rank adaptors to specifically enhance the under-represented high-frequency abilities of advanced diffusion models. Compared with progressive distillation, HiPA achieves much better performance in one-step text-to-image generation.
• Score: 47.43155993432259
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have revolutionized text-to-image generation, but their real-world applications are hampered by the extensive time needed for hundreds of diffusion steps. Although progressive distillation has been proposed to speed up diffusion sampling to 2-8 steps, it still falls short in one-step generation, and necessitates training multiple student models, which is highly parameter-extensive and time-consuming. To overcome these limitations, we introduce High-frequency-Promoting Adaptation (HiPA), a parameter-efficient approach to enable one-step text-to-image diffusion. Grounded in the insight that high-frequency information is essential but highly lacking in one-step diffusion, HiPA focuses on training one-step, low-rank adaptors to specifically enhance the under-represented high-frequency abilities of advanced diffusion models. The learned adaptors empower these diffusion models to generate high-quality images in just a single step. Compared with progressive distillation, HiPA achieves much better performance in one-step text-to-image generation (37.3 $\rightarrow$ 23.8 in FID-5k on MS-COCO 2017) and 28.6x training speed-up (108.8 $\rightarrow$ 3.8 A100 GPU days), requiring only 0.04% training parameters (7,740 million $\rightarrow$ 3.3 million). We also demonstrate HiPA's effectiveness in text-guided image editing, inpainting and super-resolution tasks, where our adapted models consistently deliver high-quality outputs in just one diffusion step. The source code will be released.

Related papers

• One-Step Diffusion Model for Image Motion-Deblurring [85.76149042561507]
  We propose a one-step diffusion model for deblurring (OSDD), a novel framework that reduces the denoising process to a single step. To tackle fidelity loss in diffusion models, we introduce an enhanced variational autoencoder (eVAE), which improves structural restoration. Our method achieves strong performance on both full and no-reference metrics.
  arXiv  Detail & Related papers  (2025-03-09T09:39:57Z)
• One Diffusion Step to Real-World Super-Resolution via Flow Trajectory Distillation [60.54811860967658]
  FluxSR is a novel one-step diffusion Real-ISR based on flow matching models. First, we introduce Flow Trajectory Distillation (FTD) to distill a multi-step flow matching model into a one-step Real-ISR. Second, to improve image realism and address high-frequency artifact issues in generated images, we propose TV-LPIPS as a perceptual loss.
  arXiv  Detail & Related papers  (2025-02-04T04:11:29Z)
• Unleashing the Power of One-Step Diffusion based Image Super-Resolution via a Large-Scale Diffusion Discriminator [81.81748032199813]
  Diffusion models have demonstrated excellent performance for real-world image super-resolution (Real-ISR) We propose a new One-Step textbfDiffusion model with a larger-scale textbfDiscriminator for SR. Our discriminator is able to distill noisy features from any time step of diffusion models in the latent space.
  arXiv  Detail & Related papers  (2024-10-05T16:41:36Z)
• Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generation [2.693650249239372]
  We propose an efficient, fast, and versatile distillation method to accelerate the generation of pre-trained diffusion models: Flash Diffusion. The method reaches state-of-the-art performances in terms of FID and CLIP-Score for few steps image generation on the COCO2014 and COCO 2017 datasets. The versatility of the method is also exposed across several tasks such as text-to-image, inpainting, face-swapping, super-resolution and using different backbones such as UNet-based denoisers (SD1.5, SDXL) or DiT (Pixart-$alpha$
  arXiv  Detail & Related papers  (2024-06-04T14:23:27Z)
• PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher [55.22994720855957]
  PaGoDA is a novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distilling the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a $64times$ reduced cost in training its diffusion model on 8x downsampled data. PaGoDA's pipeline can be applied directly in the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models.
  arXiv  Detail & Related papers  (2024-05-23T17:39:09Z)
• SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions [5.100085108873068]
  We present two models, SDXS-512 and SDXS-1024, achieving inference speeds of approximately 100 FPS (30x faster than SD v1.5) and 30 FPS (60x faster than SDXL) on a single GPU. Our training approach offers promising applications in image-conditioned control, facilitating efficient image-to-image translation.
  arXiv  Detail & Related papers  (2024-03-25T11:16:23Z)
• Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation [24.236841051249243]
  Distillation methods aim to shift the model from many-shot to single-step inference. We introduce Latent Adversarial Diffusion Distillation (LADD), a novel distillation approach overcoming the limitations of ADD. In contrast to pixel-based ADD, LADD utilizes generative features from pretrained latent diffusion models.
  arXiv  Detail & Related papers  (2024-03-18T17:51:43Z)
• One-Step Diffusion Distillation via Deep Equilibrium Models [64.11782639697883]
  We introduce a simple yet effective means of distilling diffusion models directly from initial noise to the resulting image. Our method enables fully offline training with just noise/image pairs from the diffusion model. We demonstrate that the DEQ architecture is crucial to this capability, as GET matches a $5times$ larger ViT in terms of FID scores.
  arXiv  Detail & Related papers  (2023-12-12T07:28:40Z)
• One-step Diffusion with Distribution Matching Distillation [54.723565605974294]
  We introduce Distribution Matching Distillation (DMD), a procedure to transform a diffusion model into a one-step image generator. We enforce the one-step image generator match the diffusion model at distribution level, by minimizing an approximate KL divergence. Our method outperforms all published few-step diffusion approaches, reaching 2.62 FID on ImageNet 64x64 and 11.49 FID on zero-shot COCO-30k.
  arXiv  Detail & Related papers  (2023-11-30T18:59:20Z)
• ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models [59.90959789767886]
  We show that optimizing consistency training loss minimizes the Wasserstein distance between target and generated distributions. By incorporating a discriminator into the consistency training framework, our method achieves improved FID scores on CIFAR10 and ImageNet 64$times$64 and LSUN Cat 256$times$256 datasets.
  arXiv  Detail & Related papers  (2023-11-23T16:49:06Z)
• SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds [88.06788636008051]
  Text-to-image diffusion models can create stunning images from natural language descriptions that rival the work of professional artists and photographers. These models are large, with complex network architectures and tens of denoising iterations, making them computationally expensive and slow to run. We present a generic approach that unlocks running text-to-image diffusion models on mobile devices in less than $2$ seconds.
  arXiv  Detail & Related papers  (2023-06-01T17:59:25Z)
• Wavelet Diffusion Models are fast and scalable Image Generators [3.222802562733787]
  Diffusion models are a powerful solution for high-fidelity image generation, which exceeds GANs in quality in many circumstances. Recent DiffusionGAN method significantly decreases the models' running time by reducing the number of sampling steps from thousands to several, but their speeds still largely lag behind the GAN counterparts. This paper aims to reduce the speed gap by proposing a novel wavelet-based diffusion scheme. We extract low-and-high frequency components from both image and feature levels via wavelet decomposition and adaptively handle these components for faster processing while maintaining good generation quality.
  arXiv  Detail & Related papers  (2022-11-29T12:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.