Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion

• URL: http://arxiv.org/abs/2410.19324v1
• Date: Fri, 25 Oct 2024 06:20:06 GMT
• Title: Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion
• Authors: Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, Tim Salimans,
• Abstract summary: We show that pixel-space models can in fact be very competitive to latent approaches both in quality and efficiency. We present a simple recipe for scaling end-to-end pixel-space diffusion models to high resolutions.
• Score: 34.70370851239368
• License:
• Abstract: Latent diffusion models have become the popular choice for scaling up diffusion models for high resolution image synthesis. Compared to pixel-space models that are trained end-to-end, latent models are perceived to be more efficient and to produce higher image quality at high resolution. Here we challenge these notions, and show that pixel-space models can in fact be very competitive to latent approaches both in quality and efficiency, achieving 1.5 FID on ImageNet512 and new SOTA results on ImageNet128 and ImageNet256. We present a simple recipe for scaling end-to-end pixel-space diffusion models to high resolutions. 1: Use the sigmoid loss (Kingma & Gao, 2023) with our prescribed hyper-parameters. 2: Use our simplified memory-efficient architecture with fewer skip-connections. 3: Scale the model to favor processing the image at high resolution with fewer parameters, rather than using more parameters but at a lower resolution. When combining these three steps with recently proposed tricks like guidance intervals, we obtain a family of pixel-space diffusion models we call Simple Diffusion v2 (SiD2).

Related papers

• Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models [26.926712014346432]
  This paper presents innovative enhancements to diffusion models by integrating a novel multi-resolution network and time-dependent layer normalization. Our method's efficacy is demonstrated on the class-conditional ImageNet generation benchmark, setting new state-of-the-art FID scores of 1.70 on ImageNet 256 x 256 and 2.89 on ImageNet 512 x 512.
  arXiv  Detail & Related papers  (2024-06-13T17:59:58Z)
• Hierarchical Patch Diffusion Models for High-Resolution Video Generation [50.42746357450949]
  We develop deep context fusion, which propagates context information from low-scale to high-scale patches in a hierarchical manner. We also propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation.
  arXiv  Detail & Related papers  (2024-06-12T01:12:53Z)
• SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions [5.100085108873068]
  We present two models, SDXS-512 and SDXS-1024, achieving inference speeds of approximately 100 FPS (30x faster than SD v1.5) and 30 FPS (60x faster than SDXL) on a single GPU. Our training approach offers promising applications in image-conditioned control, facilitating efficient image-to-image translation.
  arXiv  Detail & Related papers  (2024-03-25T11:16:23Z)
• Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation [112.08287900261898]
  This paper proposes a novel self-cascade diffusion model for rapid adaptation to higher-resolution image and video generation. Our approach achieves a 5X training speed-up and requires only an additional 0.002M tuning parameters. Experiments demonstrate that our approach can quickly adapt to higher resolution image and video synthesis by fine-tuning for just 10k steps, with virtually no additional inference time.
  arXiv  Detail & Related papers  (2024-02-16T07:48:35Z)
• KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis [52.42320594388199]
  We present three key practices in building an efficient text-to-image model. Based on these findings, we build two types of efficient text-to-image models, called KOALA-Turbo &-Lightning. Unlike SDXL, our KOALA models can generate 1024px high-resolution images on consumer-grade GPUs with 8GB of VRAMs (3060Ti)
  arXiv  Detail & Related papers  (2023-12-07T02:46:18Z)
• Simple diffusion: End-to-end diffusion for high resolution images [27.47227724865238]
  This paper aims to improve denoising diffusion for high resolution images while keeping the model as simple as possible. The four main findings are: 1) the noise schedule should be adjusted for high resolution images, 2) It is sufficient to scale only a particular part of the architecture, 3) dropout should be added at specific locations in the architecture, and 4) downsampling is an effective strategy to avoid high resolution feature maps.
  arXiv  Detail & Related papers  (2023-01-26T13:35:02Z)
• Magic3D: High-Resolution Text-to-3D Content Creation [78.40092800817311]
  DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF) In this paper, we address these limitations by utilizing a two-stage optimization framework. Our method, dubbed Magic3D, can create high quality 3D mesh models in 40 minutes, which is 2x faster than DreamFusion.
  arXiv  Detail & Related papers  (2022-11-18T18:59:59Z)
• On Distillation of Guided Diffusion Models [94.95228078141626]
  We propose an approach to distilling classifier-free guided diffusion models into models that are fast to sample from. For standard diffusion models trained on the pixelspace, our approach is able to generate images visually comparable to that of the original model. For diffusion models trained on the latent-space (e.g., Stable Diffusion), our approach is able to generate high-fidelity images using as few as 1 to 4 denoising steps.
  arXiv  Detail & Related papers  (2022-10-06T18:03:56Z)
• Cascaded Diffusion Models for High Fidelity Image Generation [53.57766722279425]
  We show that cascaded diffusion models are capable of generating high fidelity images on the class-conditional ImageNet generation challenge. A cascaded diffusion model comprises a pipeline of multiple diffusion models that generate images of increasing resolution. We find that the sample quality of a cascading pipeline relies crucially on conditioning augmentation.
  arXiv  Detail & Related papers  (2021-05-30T17:14:52Z)
• not-so-BigGAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution [23.15896056344987]
  Nsb-GAN is a simple yet cost-effective two-step training framework for deep generative models. Wavelet-based down-sampling method preserves more structural information than pixel-based methods. On ImageNet 512x512, our model achieves a Fr'echet Inception Distance (FID) of 10.59 -- beating the baseline BigGAN model.
  arXiv  Detail & Related papers  (2020-09-09T17:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.