Don't drop your samples! Coherence-aware training benefits Conditional diffusion

• URL: http://arxiv.org/abs/2405.20324v1
• Date: Thu, 30 May 2024 17:57:26 GMT
• Title: Don't drop your samples! Coherence-aware training benefits Conditional diffusion
• Authors: Nicolas Dufour, Victor Besnier, Vicky Kalogeiton, David Picard,
• Abstract summary: Coherence-Aware Diffusion (CAD) is a novel method that integrates coherence in conditional information into diffusion models. We show that CAD is theoretically sound and empirically effective on various conditional generation tasks.
• Score: 17.349357521783062
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conditional diffusion models are powerful generative models that can leverage various types of conditional information, such as class labels, segmentation masks, or text captions. However, in many real-world scenarios, conditional information may be noisy or unreliable due to human annotation errors or weak alignment. In this paper, we propose the Coherence-Aware Diffusion (CAD), a novel method that integrates coherence in conditional information into diffusion models, allowing them to learn from noisy annotations without discarding data. We assume that each data point has an associated coherence score that reflects the quality of the conditional information. We then condition the diffusion model on both the conditional information and the coherence score. In this way, the model learns to ignore or discount the conditioning when the coherence is low. We show that CAD is theoretically sound and empirically effective on various conditional generation tasks. Moreover, we show that leveraging coherence generates realistic and diverse samples that respect conditional information better than models trained on cleaned datasets where samples with low coherence have been discarded.

Related papers

• Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data [74.2507346810066]
  Ambient diffusion is a recently proposed framework for training diffusion models using corrupted data. We present the first framework for training diffusion models that provably sample from the uncorrupted distribution given only noisy training data.
  arXiv  Detail & Related papers  (2024-03-20T14:22:12Z)
• Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theory [87.00653989457834]
  Conditional diffusion models serve as the foundation of modern image synthesis and find extensive application in fields like computational biology and reinforcement learning. Despite the empirical success, theory of conditional diffusion models is largely missing. This paper bridges the gap by presenting a sharp statistical theory of distribution estimation using conditional diffusion models.
  arXiv  Detail & Related papers  (2024-03-18T17:08:24Z)
• Zero-Shot Conditioning of Score-Based Diffusion Models by Neuro-Symbolic Constraints [1.1826485120701153]
  We propose a method that, given a pre-trained unconditional score-based generative model, samples from the conditional distribution under arbitrary logical constraints. We show how to manipulate the learned score in order to sample from an un-normalized distribution conditional on a user-defined constraint. We define a flexible and numerically stable neuro-symbolic framework for encoding soft logical constraints.
  arXiv  Detail & Related papers  (2023-08-31T08:25:47Z)
• PriSampler: Mitigating Property Inference of Diffusion Models [6.5990719141691825]
  This work systematically presents the first privacy study about property inference attacks against diffusion models. We propose a new model-agnostic plug-in method PriSampler to infer the risks of the property inference of diffusion models.
  arXiv  Detail & Related papers  (2023-06-08T14:05:06Z)
• Conditional Generation from Unconditional Diffusion Models using Denoiser Representations [94.04631421741986]
  We propose adapting pre-trained unconditional diffusion models to new conditions using the learned internal representations of the denoiser network. We show that augmenting the Tiny ImageNet training set with synthetic images generated by our approach improves the classification accuracy of ResNet baselines by up to 8%.
  arXiv  Detail & Related papers  (2023-06-02T20:09:57Z)
• ChiroDiff: Modelling chirographic data with Diffusion Models [132.5223191478268]
  We introduce a powerful model-class namely "Denoising Diffusion Probabilistic Models" or DDPMs for chirographic data. Our model named "ChiroDiff", being non-autoregressive, learns to capture holistic concepts and therefore remains resilient to higher temporal sampling rate.
  arXiv  Detail & Related papers  (2023-04-07T15:17:48Z)
• Visual Chain-of-Thought Diffusion Models [15.547439887203613]
  We propose to close the gap between conditional and unconditional models using a two-stage sampling procedure. Doing so lets us leverage the power of conditional diffusion models on the unconditional generation task, which we show improves FID by 25-50% compared to standard unconditional generation.
  arXiv  Detail & Related papers  (2023-03-28T17:53:06Z)
• ShiftDDPMs: Exploring Conditional Diffusion Models by Shifting Diffusion Trajectories [144.03939123870416]
  We propose a novel conditional diffusion model by introducing conditions into the forward process. We use extra latent space to allocate an exclusive diffusion trajectory for each condition based on some shifting rules. We formulate our method, which we call textbfShiftDDPMs, and provide a unified point of view on existing related methods.
  arXiv  Detail & Related papers  (2023-02-05T12:48:21Z)
• Why Are Conditional Generative Models Better Than Unconditional Ones? [36.870497480570776]
  We propose self-conditioned diffusion models (SCDM), which is trained conditioned on indices clustered by the k-means algorithm on the features extracted by a model pre-trained in a self-supervised manner. SCDM significantly achieves the unconditional model across various datasets and a record-breaking FID of 3.94 on ImageNet 64x64 without labels.
  arXiv  Detail & Related papers  (2022-12-01T08:44:21Z)
• Collapse by Conditioning: Training Class-conditional GANs with Limited Data [109.30895503994687]
  We propose a training strategy for conditional GANs (cGANs) that effectively prevents the observed mode-collapse by leveraging unconditional learning. Our training strategy starts with an unconditional GAN and gradually injects conditional information into the generator and the objective function. The proposed method for training cGANs with limited data results not only in stable training but also in generating high-quality images.
  arXiv  Detail & Related papers  (2022-01-17T18:59:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.