Conditional Distribution Modelling for Few-Shot Image Synthesis with Diffusion Models

• URL: http://arxiv.org/abs/2404.16556v2
• Date: Mon, 29 Apr 2024 03:09:33 GMT
• Title: Conditional Distribution Modelling for Few-Shot Image Synthesis with Diffusion Models
• Authors: Parul Gupta, Munawar Hayat, Abhinav Dhall, Thanh-Toan Do,
• Abstract summary: Few-shot image synthesis entails generating diverse and realistic images of novel categories using only a few example images. We propose Conditional Distribution Modelling (CDM) -- a framework which effectively utilizes Diffusion models for few-shot image generation.
• Score: 29.821909424996015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot image synthesis entails generating diverse and realistic images of novel categories using only a few example images. While multiple recent efforts in this direction have achieved impressive results, the existing approaches are dependent only upon the few novel samples available at test time in order to generate new images, which restricts the diversity of the generated images. To overcome this limitation, we propose Conditional Distribution Modelling (CDM) -- a framework which effectively utilizes Diffusion models for few-shot image generation. By modelling the distribution of the latent space used to condition a Diffusion process, CDM leverages the learnt statistics of the training data to get a better approximation of the unseen class distribution, thereby removing the bias arising due to limited number of few shot samples. Simultaneously, we devise a novel inversion based optimization strategy that further improves the approximated unseen class distribution, and ensures the fidelity of the generated samples to the unseen class. The experimental results on four benchmark datasets demonstrate the effectiveness of our proposed CDM for few-shot generation.

Related papers

• Rejection Sampling IMLE: Designing Priors for Better Few-Shot Image Synthesis [7.234618871984921]
  An emerging area of research aims to learn deep generative models with limited training data. We propose RS-IMLE, a novel approach that changes the prior distribution used for training. This leads to substantially higher quality image generation compared to existing GAN and IMLE-based methods.
  arXiv  Detail & Related papers  (2024-09-26T00:19:42Z)
• Few-Shot Image Generation by Conditional Relaxing Diffusion Inversion [37.18537753482751]
  Conditional Diffusion Relaxing Inversion (CRDI) is designed to enhance distribution diversity in synthetic image generation. CRDI does not rely on fine-tuning based on only a few samples. It focuses on reconstructing each target image instance and expanding diversity through few-shot learning.
  arXiv  Detail & Related papers  (2024-07-09T21:58:26Z)
• EM Distillation for One-step Diffusion Models [65.57766773137068]
  We propose a maximum likelihood-based approach that distills a diffusion model to a one-step generator model with minimal loss of quality. We develop a reparametrized sampling scheme and a noise cancellation technique that together stabilizes the distillation process.
  arXiv  Detail & Related papers  (2024-05-27T05:55:22Z)
• TC-DiffRecon: Texture coordination MRI reconstruction method based on diffusion model and modified MF-UNet method [2.626378252978696]
  We propose a novel diffusion model-based MRI reconstruction method, named TC-DiffRecon, which does not rely on a specific acceleration factor for training. We also suggest the incorporation of the MF-UNet module, designed to enhance the quality of MRI images generated by the model.
  arXiv  Detail & Related papers  (2024-02-17T13:09:00Z)
• Training Class-Imbalanced Diffusion Model Via Overlap Optimization [55.96820607533968]
  Diffusion models trained on real-world datasets often yield inferior fidelity for tail classes. Deep generative models, including diffusion models, are biased towards classes with abundant training images. We propose a method based on contrastive learning to minimize the overlap between distributions of synthetic images for different classes.
  arXiv  Detail & Related papers  (2024-02-16T16:47:21Z)
• Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis [62.07413805483241]
  Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
  arXiv  Detail & Related papers  (2023-09-30T02:03:22Z)
• Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation [18.956306942099097]
  Conditional diffusion models (CDM) is capable of generating images subject to specific distributions. We utilize category-aware semantic information underlied in CDM to get the prediction mask of the target object. Our method outperforms state-of-the-art CAM and diffusion model methods on two public medical image segmentation datasets.
  arXiv  Detail & Related papers  (2023-06-06T17:29:26Z)
• Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC [102.64648158034568]
  diffusion models have quickly become the prevailing approach to generative modeling in many domains. We propose an energy-based parameterization of diffusion models which enables the use of new compositional operators. We find these samplers lead to notable improvements in compositional generation across a wide set of problems.
  arXiv  Detail & Related papers  (2023-02-22T18:48:46Z)
• ShiftDDPMs: Exploring Conditional Diffusion Models by Shifting Diffusion Trajectories [144.03939123870416]
  We propose a novel conditional diffusion model by introducing conditions into the forward process. We use extra latent space to allocate an exclusive diffusion trajectory for each condition based on some shifting rules. We formulate our method, which we call textbfShiftDDPMs, and provide a unified point of view on existing related methods.
  arXiv  Detail & Related papers  (2023-02-05T12:48:21Z)
• Fast Inference in Denoising Diffusion Models via MMD Finetuning [23.779985842891705]
  We present MMD-DDM, a novel method for fast sampling of diffusion models. Our approach is based on the idea of using the Maximum Mean Discrepancy (MMD) to finetune the learned distribution with a given budget of timesteps. Our findings show that the proposed method is able to produce high-quality samples in a fraction of the time required by widely-used diffusion models.
  arXiv  Detail & Related papers  (2023-01-19T09:48:07Z)
• On Distillation of Guided Diffusion Models [94.95228078141626]
  We propose an approach to distilling classifier-free guided diffusion models into models that are fast to sample from. For standard diffusion models trained on the pixelspace, our approach is able to generate images visually comparable to that of the original model. For diffusion models trained on the latent-space (e.g., Stable Diffusion), our approach is able to generate high-fidelity images using as few as 1 to 4 denoising steps.
  arXiv  Detail & Related papers  (2022-10-06T18:03:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.