An exact information theory of generalization phase transitions in Bayesian diffusion models

• URL: http://arxiv.org/abs/2607.08041v1
• Date: Thu, 09 Jul 2026 01:33:11 GMT
• Title: An exact information theory of generalization phase transitions in Bayesian diffusion models
• Abstract summary: We show how diffusion models circumvent the curse of dimensionality to learn over high dimensional spaces from a finite training set, instead of memorizing it.<n>A BIRD model time-reverses diffusion by inferring which past training sample produced its current restricted observation.<n>We find that generation proceeds near the edge of memorization: both spatially local BIRD models and early-training diffusion models track the memorization-generalization phase boundary.
• Score: 11.82439668395711
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How diffusion models circumvent the curse of dimensionality to learn complex distributions over high dimensional spaces from a finite training set, instead of memorizing it, remains a fundamental mystery. To address this, we introduce analytically tractable Bayesian information restricted diffusion (BIRD) models, in which each pixel observes restricted information about noisy data. A BIRD model time-reverses diffusion by inferring which past training sample produced its current restricted observation using the Bayesian posterior. This model class generalizes existing analytical diffusion models that use spatially local information restriction. We show that spatially local BIRD models closely approximate trained diffusion models \textit{early in training}, across different architectures such as UNets and DiTs. Under minimal assumptions on the data distribution, we identify an information-theoretic phase boundary between memorization and generalization in the joint space of amount of training data, time in the reverse generative process, and amount of information restriction: a BIRD model memorizes when the mutual information between its restricted noisy observations and the training data exceeds the log number of training points, and it generalizes otherwise. Experiments across a range of datasets confirm our theoretically predicted location for the transition. We find that generation proceeds near the edge of memorization: both spatially local BIRD models and early-training diffusion models track the memorization-generalization phase boundary by increasingly restricting information over time. Overall, our results reveal a fundamental role for information restriction in generative AI to circumvent the curse of dimensionality.

Related papers

• Impute-EM: Native Mixed-State Diffusion Models for Heterogeneous Data Imputation [71.74203645336385]
  Impute-EM alternates between imputing missing entries with the current model and refitting a diffusion backbone on completed data.<n>We characterize the update and show that the observed mask-indexed marginals match the targets at the limit.
  arXiv  Detail & Related papers  (2026-09-14T09:39:13Z)
• Local Coverage Governs Memorization in Diffusion Models [10.085837353469737]
  We show that memorization is governed by emphlocal data coverage.<n>We show that classes with higher intra-class sparsity (and thus lower local coverage) are more strongly memorized.
  arXiv  Detail & Related papers  (2026-06-12T12:25:13Z)
• The Principles of Diffusion Models [81.12042238390075]
  Diffusion modeling starts by defining a forward process that gradually corrupts data into noise.<n>The goal is to learn a reverse process that transforms noise back into data while recovering the same intermediates.<n>The score-based view, rooted in energy-based modeling, learns the gradient of the evolving data distribution.<n>The flow-based view, related to normalizing flows, treats generation as following a smooth path that moves samples from noise to data.
  arXiv  Detail & Related papers  (2025-10-24T02:29:02Z)
• DS-Diffusion: Data Style-Guided Diffusion Model for Time-Series Generation [3.7098771725459336]
  We propose a data style-guided diffusion model (DS-Diffusion) for time series generation tasks.<n>The DS-Diffusion avoids retraining the entire framework to introduce conditional guidance.<n>The generated samples can clearly indicate the data style from which they originate.
  arXiv  Detail & Related papers  (2025-09-23T03:06:39Z)
• On the Edge of Memorization in Diffusion Models [25.927892368310868]
  We introduce a scientific and mathematical "laboratory" for investigating memorization and generalization in practical diffusion models.<n>Our work provides an analytically tractable and practically meaningful setting for future theoretical and empirical investigations.
  arXiv  Detail & Related papers  (2025-08-25T05:56:05Z)
• Memorization to Generalization: Emergence of Diffusion Models from Associative Memory [27.677278702224687]
  We study the memorization-generalization phenomenon in diffusion models via the lens of AMs.<n>Our findings provide: a novel perspective on the memorization-generalization phenomenon in diffusion models via the lens of AMs.
  arXiv  Detail & Related papers  (2025-05-27T21:20:57Z)
• Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models [56.032091696552094]
  Generalization in natural data domains is progressively achieved during training before the onset of memorization.<n>Generalization vs. memorization is then best understood as a competition between time scales.<n>We show that this phenomenology is recovered in diffusion models learning a simple probabilistic context-free grammar with random rules.
  arXiv  Detail & Related papers  (2025-05-22T17:40:08Z)
• Constrained Diffusion Models via Dual Training [80.03953599062365]
  Diffusion processes are prone to generating samples that reflect biases in a training dataset. We develop constrained diffusion models by imposing diffusion constraints based on desired distributions. We show that our constrained diffusion models generate new data from a mixture data distribution that achieves the optimal trade-off among objective and constraints.
  arXiv  Detail & Related papers  (2024-08-27T14:25:42Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• A Temporally Disentangled Contrastive Diffusion Model for Spatiotemporal Imputation [35.46631415365955]
  We introduce a conditional diffusion framework called C$2$TSD, which incorporates disentangled temporal (trend and seasonality) representations as conditional information. Our experiments on three real-world datasets demonstrate the superior performance of our approach compared to a number of state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-18T11:59:04Z)
• Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models [63.1637853118899]
  We propose the first unsupervised and learning-based method to identify interpretable directions in h-space of pre-trained diffusion models. We employ a shift control module that works on h-space of pre-trained diffusion models to manipulate a sample into a shifted version of itself. By jointly optimizing them, the model will spontaneously discover disentangled and interpretable directions.
  arXiv  Detail & Related papers  (2023-10-15T18:44:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.