Capturing Conditional Dependence via Auto-regressive Diffusion Models

• URL: http://arxiv.org/abs/2504.21314v1
• Date: Wed, 30 Apr 2025 04:57:12 GMT
• Title: Capturing Conditional Dependence via Auto-regressive Diffusion Models
• Authors: Xunpeng Huang, Yujin Han, Difan Zou, Yian Ma, Tong Zhang,
• Abstract summary: We study the efficacy of auto-regressive (AR) diffusion models for capturing conditional dependence structures in the data.<n>Our theoretical findings indicate that, compared with typical diffusion models, the AR variant produces samples with a reduced gap in approximating the data conditional distribution.<n>We also provide empirical results showing that when there is clear conditional dependence structure in the data, the AR diffusion models captures such structure, whereas vanilla DDPM fails to do so.
• Score: 24.26847446193959
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have demonstrated appealing performance in both image and video generation. However, many works discover that they struggle to capture important, high-level relationships that are present in the real world. For example, they fail to learn physical laws from data, and even fail to understand that the objects in the world exist in a stable fashion. This is due to the fact that important conditional dependence structures are not adequately captured in the vanilla diffusion models. In this work, we initiate an in-depth study on strengthening the diffusion model to capture the conditional dependence structures in the data. In particular, we examine the efficacy of the auto-regressive (AR) diffusion models for such purpose and develop the first theoretical results on the sampling error of AR diffusion models under (possibly) the mildest data assumption. Our theoretical findings indicate that, compared with typical diffusion models, the AR variant produces samples with a reduced gap in approximating the data conditional distribution. On the other hand, the overall inference time of the AR-diffusion models is only moderately larger than that for the vanilla diffusion models, making them still practical for large scale applications. We also provide empirical results showing that when there is clear conditional dependence structure in the data, the AR diffusion models captures such structure, whereas vanilla DDPM fails to do so. On the other hand, when there is no obvious conditional dependence across patches of the data, AR diffusion does not outperform DDPM.

Related papers

• Continuous Diffusion Model for Language Modeling [57.396578974401734]
  Existing continuous diffusion models for discrete data have limited performance compared to discrete approaches. We propose a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution.
  arXiv  Detail & Related papers  (2025-02-17T08:54:29Z)
• Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models [22.39558434131574]
  Existing data attribution methods for diffusion models typically quantify the contribution of a training sample.<n>We argue that the direct usage of diffusion loss cannot represent such a contribution accurately due to the calculation of diffusion loss.<n>We propose Diffusion Attribution Score (textitDAS) to measure the direct comparison between predicted distributions with an attribution score.
  arXiv  Detail & Related papers  (2024-10-24T10:58:17Z)
• Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions [6.9408143976091745]
  Denoising Diffusion Probabilistic Models (DDPM) are powerful state-of-the-art methods used to generate synthetic data from high-dimensional data distributions.<n>We study DDPMs under the manifold hypothesis and prove that they achieve rates independent of the ambient dimension in terms of score learning.<n>In terms of sampling complexity, we obtain rates independent of the ambient dimension w.r.t. the Kullback-Leibler divergence, and $O(sqrtD)$ w.r.t. the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-09-27T14:57:18Z)
• An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization [59.63880337156392]
  Diffusion models have achieved tremendous success in computer vision, audio, reinforcement learning, and computational biology. Despite the significant empirical success, theory of diffusion models is very limited. This paper provides a well-rounded theoretical exposure for stimulating forward-looking theories and methods of diffusion models.
  arXiv  Detail & Related papers  (2024-04-11T14:07:25Z)
• Guided Diffusion from Self-Supervised Diffusion Features [49.78673164423208]
  Guidance serves as a key concept in diffusion models, yet its effectiveness is often limited by the need for extra data annotation or pretraining. We propose a framework to extract guidance from, and specifically for, diffusion models.
  arXiv  Detail & Related papers  (2023-12-14T11:19:11Z)
• The Emergence of Reproducibility and Generalizability in Diffusion Models [10.188731323681575]
  Given the same starting noise input and a deterministic sampler, different diffusion models often yield remarkably similar outputs. We show that diffusion models are learning distinct distributions affected by the training data size. This valuable property generalizes to many variants of diffusion models, including those for conditional use, solving inverse problems, and model fine-tuning.
  arXiv  Detail & Related papers  (2023-10-08T19:02:46Z)
• Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models [76.46246743508651]
  We show that current diffusion models actually have an expressive bottleneck in backward denoising. We introduce soft mixture denoising (SMD), an expressive and efficient model for backward denoising.
  arXiv  Detail & Related papers  (2023-09-25T12:03:32Z)
• On Error Propagation of Diffusion Models [77.91480554418048]
  We develop a theoretical framework to mathematically formulate error propagation in the architecture of DMs. We apply the cumulative error as a regularization term to reduce error propagation. Our proposed regularization reduces error propagation, significantly improves vanilla DMs, and outperforms previous baselines.
  arXiv  Detail & Related papers  (2023-08-09T15:31:17Z)
• Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations [6.2284442126065525]
  We study the problem mathematically for two limiting scenarios: the zero diffusion (ODE) case and the large diffusion case. Our findings indicate that when the perturbation occurs at the end of the generative process, the ODE model outperforms the SDE model with a large diffusion coefficient.
  arXiv  Detail & Related papers  (2023-06-03T09:27:15Z)
• Diffusion Models are Minimax Optimal Distribution Estimators [49.47503258639454]
  We provide the first rigorous analysis on approximation and generalization abilities of diffusion modeling. We show that when the true density function belongs to the Besov space and the empirical score matching loss is properly minimized, the generated data distribution achieves the nearly minimax optimal estimation rates.
  arXiv  Detail & Related papers  (2023-03-03T11:31:55Z)
• Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance [95.12230117950232]
  We show that a common latent space emerges from two diffusion models trained independently on related domains. Applying CycleDiffusion to text-to-image diffusion models, we show that large-scale text-to-image diffusion models can be used as zero-shot image-to-image editors.
  arXiv  Detail & Related papers  (2022-10-11T15:53:52Z)
• Non-Uniform Diffusion Models [0.8602553195689513]
  We show that non-uniform diffusion leads to multi-scale diffusion models which have similar structure to this of multi-scale normalizing flows. We experimentally find that in the same or less training time, the multi-scale diffusion model achieves better FID score than the standard uniform diffusion model. We also show that non-uniform diffusion leads to a novel estimator for the conditional score function which achieves on par performance with the state-of-the-art conditional denoising estimator.
  arXiv  Detail & Related papers  (2022-07-20T09:59:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.