Memorisation, convergence and generalisation in generative models

• URL: http://arxiv.org/abs/2605.21402v1
• Date: Wed, 20 May 2026 17:00:37 GMT
• Title: Memorisation, convergence and generalisation in generative models
• Abstract summary: We show that generalisation in generative models decomposes into at least two distinct objectives.<n>These objectives correspond to two different distances between true and learnt data distribution.
• Score: 13.652405522895128
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative neural networks learn how to produce highly realistic images from a large, but finite number of examples - or do they simply memorise their training set? To settle this question, Kadkhodaie, Guth, Simoncelli and Mallat (ICLR '24) trained diffusion models independently on disjoint subsets of a dataset and showed that they converge to nearly the same density when the number of training images is large enough. This result raises two basic questions: how much data do you need for convergence, and what does convergence capture about learning the data distribution? Here, we address these questions by providing an exact analytical characterisation of the transition from memorisation to generalisation in linear generative models. We find that these models memorise at small load, while convergence emerges continuously when the number of samples is linear in the input dimension. Strikingly, we find that convergence is insensitive to recovery of the principal latent factors of the data, which are recovered in a sharp transition. After extending our approach to data with power-law spectra, we find the same distinction between convergence and latent recovery in our experiments with convolutional denoisers and in the data of Kadkhodaie et al. We thus show that generalisation in generative models decomposes into at least two distinct objectives: matching the bulk of the data distribution and recovering the principal latent factors. These objectives correspond to two different distances between true and learnt data distribution, and only the first one is captured by convergence.

Related papers

• Impute-EM: Native Mixed-State Diffusion Models for Heterogeneous Data Imputation [71.74203645336385]
  Impute-EM alternates between imputing missing entries with the current model and refitting a diffusion backbone on completed data.<n>We characterize the update and show that the observed mask-indexed marginals match the targets at the limit.
  arXiv  Detail & Related papers  (2026-09-14T09:39:13Z)
• Multi-Objective Learning for Diffusion Models: A Statistical Theory under Semi-Supervised Learning [83.65494573724904]
  We develop a principled MOL framework for diffusion models with limited data.<n>We propose a two-stage training procedure that first fits lightweight specialist models from limited paired data.<n>We establish bounds generalization showing that the required number of paired samples only depends on the complexity of the specialist model classes.
  arXiv  Detail & Related papers  (2026-05-24T18:19:40Z)
• Impugan: Learning Conditional Generative Models for Robust Data Imputation [0.21847754147782886]
  Impugan is a conditional Generative Adversarial Network (cGAN) for imputing missing values and integrating heterogeneous datasets.<n>During inference, the generator reconstructs missing entries from available features, and the discriminator enforces realism by distinguishing true from imputed data.
  arXiv  Detail & Related papers  (2025-12-05T18:46:33Z)
• Data-Dependent Smoothing for Protein Discovery with Walk-Jump Sampling [7.278972126771497]
  Diffusion models have emerged as a powerful class of generative models by learning to iteratively reverse the noising process.<n>Their ability to generate high-quality samples has extended beyond high-dimensional image data to other complex domains such as proteins.<n>We introduce a Data-Dependent Smoothing Walk-Jump framework that employs kernel density estimation (KDE) as a preprocessing step to estimate the noise scale $sigma$ for each data point.<n>By incorporating local data geometry into the denoising process, our method accounts for the heterogeneous distribution of protein data.
  arXiv  Detail & Related papers  (2025-09-02T08:17:59Z)
• Resolving Memorization in Empirical Diffusion Model for Manifold Data in High-Dimensional Spaces [5.716752583983991]
  When the data distribution consists of n points, empirical diffusion models tend to reproduce existing data points.<n>This work shows that the memorization issue can be solved simply by applying an inertia update at the end of the empirical diffusion simulation.<n>We demonstrate that the distribution of samples from this model approximates the true data distribution on a $C2$ manifold of dimension $d$, within a Wasserstein-1 distance of order $O(n-frac2d+4)$.
  arXiv  Detail & Related papers  (2025-05-05T09:40:41Z)
• Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts [1.9458156037869137]
  convergent learning is the degree to which neural systems arrive at similar internal representations.<n>We present a large-scale audit of convergent learning spanning dozens of vision models and thousands of layer-pair comparisons.<n>Findings fill critical gaps in our understanding of representational convergence, with implications for neuroscience and AI.
  arXiv  Detail & Related papers  (2025-02-26T00:04:24Z)
• A solvable generative model with a linear, one-step denoiser [0.0]
  We develop an analytically tractable single-step diffusion model based on a linear denoiser and present an explicit formula for the Kullback-Leibler divergence.<n>For large-scale practical diffusion models, we explain why a higher number of diffusion steps enhances production quality.
  arXiv  Detail & Related papers  (2024-11-26T19:00:01Z)
• Heat Death of Generative Models in Closed-Loop Learning [63.83608300361159]
  We study the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset. We show that, unless a sufficient amount of external data is introduced at each iteration, any non-trivial temperature leads the model to degenerate.
  arXiv  Detail & Related papers  (2024-04-02T21:51:39Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality [84.94877848357896]
  Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. We analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias and the tendency to memorize whole examples. We show substantial empirical improvements using standard sequence-to-sequence models on two widely-used compositionality datasets.
  arXiv  Detail & Related papers  (2022-11-28T17:36:41Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Binary Independent Component Analysis via Non-stationarity [7.283533791778359]
  We consider independent component analysis of binary data. We start by assuming a linear mixing model in a continuous-valued latent space, followed by a binary observation model. In stark contrast to the continuous-valued case, we prove non-identifiability of the model with few observed variables.
  arXiv  Detail & Related papers  (2021-11-30T14:23:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.