Implicit Regularisation in Diffusion Models: An Algorithm-Dependent Generalisation Analysis

• URL: http://arxiv.org/abs/2507.03756v1
• Date: Fri, 04 Jul 2025 18:07:06 GMT
• Title: Implicit Regularisation in Diffusion Models: An Algorithm-Dependent Generalisation Analysis
• Authors: Tyler Farghly, Patrick Rebeschini, George Deligiannidis, Arnaud Doucet,
• Abstract summary: We develop a theory of algorithm-dependent generalisation for high-dimensional diffusion models.<n>We derive generalisation bounds in terms of score stability, and apply our framework to several fundamental learning settings.
• Score: 44.468416523840965
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The success of denoising diffusion models raises important questions regarding their generalisation behaviour, particularly in high-dimensional settings. Notably, it has been shown that when training and sampling are performed perfectly, these models memorise training data -- implying that some form of regularisation is essential for generalisation. Existing theoretical analyses primarily rely on algorithm-independent techniques such as uniform convergence, heavily utilising model structure to obtain generalisation bounds. In this work, we instead leverage the algorithmic aspects that promote generalisation in diffusion models, developing a general theory of algorithm-dependent generalisation for this setting. Borrowing from the framework of algorithmic stability, we introduce the notion of score stability, which quantifies the sensitivity of score-matching algorithms to dataset perturbations. We derive generalisation bounds in terms of score stability, and apply our framework to several fundamental learning settings, identifying sources of regularisation. In particular, we consider denoising score matching with early stopping (denoising regularisation), sampler-wide coarse discretisation (sampler regularisation) and optimising with SGD (optimisation regularisation). By grounding our analysis in algorithmic properties rather than model structure, we identify multiple sources of implicit regularisation unique to diffusion models that have so far been overlooked in the literature.

Related papers

• Algorithm- and Data-Dependent Generalization Bounds for Score-Based Generative Models [27.78637798976204]
  Score-based generative models (SGMs) have emerged as one of the most popular classes of generative models.<n>This paper provides the first algorithmic- and data-dependent analysis for SGMs.<n>In particular, we account for the dynamics of the learning algorithm, offering new insights into the behavior of SGMs.
  arXiv  Detail & Related papers  (2025-06-04T11:33:04Z)
• Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimization [22.577751005038543]
  Federated Learning (FL) is a distributed learning approach that trains machine learning models across multiple devices.<n>FL often faces challenges due to data heterogeneity, leading to inconsistent local optima among clients.<n>This paper introduces an innovative generalization dynamics analysis framework, named as Libra, for algorithm-dependent excess risk minimization.
  arXiv  Detail & Related papers  (2024-11-25T11:43:22Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Stability and Generalization of the Decentralized Stochastic Gradient Descent Ascent Algorithm [80.94861441583275]
  We investigate the complexity of the generalization bound of the decentralized gradient descent (D-SGDA) algorithm. Our results analyze the impact of different top factors on the generalization of D-SGDA. We also balance it with the generalization to obtain the optimal convex-concave setting.
  arXiv  Detail & Related papers  (2023-10-31T11:27:01Z)
• Understanding Generalization of Federated Learning via Stability: Heterogeneity Matters [1.4502611532302039]
  Generalization performance is a key metric in evaluating machine learning models when applied to real-world applications. Generalization performance is a key metric in evaluating machine learning models when applied to real-world applications.
  arXiv  Detail & Related papers  (2023-06-06T16:12:35Z)
• Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability [107.65337427333064]
  optimization of the Area Under the Precision-Recall Curve (AUPRC) is a crucial problem for machine learning. In this work, we present the first trial in the single-dependent generalization of AUPRC optimization. Experiments on three image retrieval datasets on speak to the effectiveness and soundness of our framework.
  arXiv  Detail & Related papers  (2022-09-27T09:06:37Z)
• Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression [19.047997113063147]
  We introduce a notion called data-algorithm compatibility, which considers the generalization behavior of the entire data-dependent training trajectory. We perform a data-dependent trajectory analysis and derive a sufficient condition for compatibility in such a setting.
  arXiv  Detail & Related papers  (2022-02-12T12:42:36Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)
• Stochastic batch size for adaptive regularization in deep network optimization [63.68104397173262]
  We propose a first-order optimization algorithm incorporating adaptive regularization applicable to machine learning problems in deep learning framework. We empirically demonstrate the effectiveness of our algorithm using an image classification task based on conventional network models applied to commonly used benchmark datasets.
  arXiv  Detail & Related papers  (2020-04-14T07:54:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.