Mitigating Embedding Collapse in Diffusion Models for Categorical Data

• URL: http://arxiv.org/abs/2410.14758v1
• Date: Fri, 18 Oct 2024 09:12:33 GMT
• Title: Mitigating Embedding Collapse in Diffusion Models for Categorical Data
• Authors: Bac Nguyen, and Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Stefano Ermon, Yuki Mitsufuji,
• Abstract summary: We introduce CATDM, a continuous diffusion framework within the embedding space that stabilizes training. Experiments on benchmarks show that CATDM mitigates embedding collapse, yielding superior results on FFHQ, LSUN Churches, and LSUN Bedrooms.
• Score: 52.90687881724333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Latent diffusion models have enabled continuous-state diffusion models to handle a variety of datasets, including categorical data. However, most methods rely on fixed pretrained embeddings, limiting the benefits of joint training with the diffusion model. While jointly learning the embedding (via reconstruction loss) and the latent diffusion model (via score matching loss) could enhance performance, our analysis shows that end-to-end training risks embedding collapse, degrading generation quality. To address this issue, we introduce CATDM, a continuous diffusion framework within the embedding space that stabilizes training. We propose a novel objective combining the joint embedding-diffusion variational lower bound with a Consistency-Matching (CM) regularizer, alongside a shifted cosine noise schedule and random dropping strategy. The CM regularizer ensures the recovery of the true data distribution. Experiments on benchmarks show that CATDM mitigates embedding collapse, yielding superior results on FFHQ, LSUN Churches, and LSUN Bedrooms. In particular, CATDM achieves an FID of 6.81 on ImageNet $256\times256$ with 50 steps. It outperforms non-autoregressive models in machine translation and is on a par with previous methods in text generation.

Related papers

• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• Struggle with Adversarial Defense? Try Diffusion [8.274506117450628]
  Adrial attacks induce misclassification by introducing subtle perturbations. diffusion-based adversarial training often encounters convergence challenges and high computational expenses. We propose the Truth Maximization Diffusion (TMDC) to overcome these issues.
  arXiv  Detail & Related papers  (2024-04-12T06:52:40Z)
• Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data [74.2507346810066]
  Ambient diffusion is a recently proposed framework for training diffusion models using corrupted data. We present the first framework for training diffusion models that provably sample from the uncorrupted distribution given only noisy training data.
  arXiv  Detail & Related papers  (2024-03-20T14:22:12Z)
• Mitigating Data Consistency Induced Discrepancy in Cascaded Diffusion Models for Sparse-view CT Reconstruction [4.227116189483428]
  This study introduces a novel Cascaded Diffusion with Discrepancy Mitigation framework. It includes the low-quality image generation in latent space and the high-quality image generation in pixel space. It minimizes computational costs by moving some inference steps from pixel space to latent space.
  arXiv  Detail & Related papers  (2024-03-14T12:58:28Z)
• DiffClass: Diffusion-Based Class Incremental Learning [30.514281721324853]
  Class Incremental Learning (CIL) is challenging due to catastrophic forgetting. Recent exemplar-free CIL methods attempt to mitigate catastrophic forgetting by synthesizing previous task data. We propose a novel exemplar-free CIL method to overcome these issues.
  arXiv  Detail & Related papers  (2024-03-08T03:34:18Z)
• Fixed Point Diffusion Models [13.035518953879539]
  Fixed Point Diffusion Model (FPDM) is a novel approach to image generation that integrates the concept of fixed point solving into the framework of diffusion-based generative modeling. Our approach embeds an implicit fixed point solving layer into the denoising network of a diffusion model, transforming the diffusion process into a sequence of closely-related fixed point problems. We conduct experiments with state-of-the-art models on ImageNet, FFHQ, CelebA-HQ, and LSUN-Church, demonstrating substantial improvements in performance and efficiency.
  arXiv  Detail & Related papers  (2024-01-16T18:55:54Z)
• EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models [4.21216544443537]
  Quantization can effectively reduce model complexity, and post-training quantization (PTQ) is highly promising for compressing and accelerating diffusion models. Existing PTQ methods for diffusion models suffer from distribution mismatch issues at both calibration sample level and reconstruction output level. We propose Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models (EDA-DM) to address the above issues.
  arXiv  Detail & Related papers  (2024-01-09T14:42:49Z)
• Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models [52.1809084559048]
  We propose a novel two-stage divide-and-conquer training strategy termed TDC Training. It groups timesteps based on task similarity and difficulty, assigning highly customized denoising models to each group, thereby enhancing the performance of diffusion models. While two-stage training avoids the need to train each model separately, the total training cost is even lower than training a single unified denoising model.
  arXiv  Detail & Related papers  (2023-12-20T03:32:58Z)
• One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls [77.42510898755037]
  One More Step (OMS) is a compact network that incorporates an additional simple yet effective step during inference. OMS elevates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model parameters. Once trained, various pre-trained diffusion models with the same latent domain can share the same OMS module.
  arXiv  Detail & Related papers  (2023-11-27T12:02:42Z)
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference [60.32804641276217]
  We propose Latent Consistency Models (LCMs), enabling swift inference with minimal steps on any pre-trained LDMs. A high-quality 768 x 768 24-step LCM takes only 32 A100 GPU hours for training. We also introduce Latent Consistency Fine-tuning (LCF), a novel method that is tailored for fine-tuning LCMs on customized image datasets.
  arXiv  Detail & Related papers  (2023-10-06T17:11:58Z)
• Phasic Content Fusing Diffusion Model with Directional Distribution Consistency for Few-Shot Model Adaption [73.98706049140098]
  We propose a novel phasic content fusing few-shot diffusion model with directional distribution consistency loss. Specifically, we design a phasic training strategy with phasic content fusion to help our model learn content and style information when t is large. Finally, we propose a cross-domain structure guidance strategy that enhances structure consistency during domain adaptation.
  arXiv  Detail & Related papers  (2023-09-07T14:14:11Z)
• Low-Light Image Enhancement with Wavelet-based Diffusion Models [50.632343822790006]
  Diffusion models have achieved promising results in image restoration tasks, yet suffer from time-consuming, excessive computational resource consumption, and unstable restoration. We propose a robust and efficient Diffusion-based Low-Light image enhancement approach, dubbed DiffLL.
  arXiv  Detail & Related papers  (2023-06-01T03:08:28Z)
• GSURE-Based Diffusion Model Training with Corrupted Data [35.56267114494076]
  We propose a novel training technique for generative diffusion models based only on corrupted data. We demonstrate our technique on face images as well as Magnetic Resonance Imaging (MRI)
  arXiv  Detail & Related papers  (2023-05-22T15:27:20Z)
• Hierarchical Integration Diffusion Model for Realistic Image Deblurring [71.76410266003917]
  Diffusion models (DMs) have been introduced in image deblurring and exhibited promising performance. We propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T12:18:20Z)
• Restoration based Generative Models [0.886014926770622]
  Denoising diffusion models (DDMs) have attracted increasing attention by showing impressive synthesis quality. In this paper, we establish the interpretation of DDMs in terms of image restoration (IR) We propose a multi-scale training, which improves the performance compared to the diffusion process, by taking advantage of the flexibility of the forward process. We believe that our framework paves the way for designing a new type of flexible general generative model.
  arXiv  Detail & Related papers  (2023-02-20T00:53:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.