Selective Underfitting in Diffusion Models

• URL: http://arxiv.org/abs/2510.01378v1
• Date: Wed, 01 Oct 2025 19:06:20 GMT
• Title: Selective Underfitting in Diffusion Models
• Authors: Kiwhan Song, Jaeyeon Kim, Sitan Chen, Yilun Du, Sham Kakade, Vincent Sitzmann,
• Abstract summary: We argue that diffusion models underfit the empirical score due to training-time inductive biases.<n>Instead of underfitting the score everywhere, better diffusion models more accurately approximate the score in certain regions of input space, while underfitting it in others.<n>Our results establish that selective underfitting is essential for understanding diffusion models, yielding new, testable insights into their generalization and generative performance.
• Score: 62.51850508420274
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have emerged as the principal paradigm for generative modeling across various domains. During training, they learn the score function, which in turn is used to generate samples at inference. They raise a basic yet unsolved question: which score do they actually learn? In principle, a diffusion model that matches the empirical score in the entire data space would simply reproduce the training data, failing to generate novel samples. Recent work addresses this question by arguing that diffusion models underfit the empirical score due to training-time inductive biases. In this work, we refine this perspective, introducing the notion of selective underfitting: instead of underfitting the score everywhere, better diffusion models more accurately approximate the score in certain regions of input space, while underfitting it in others. We characterize these regions and design empirical interventions to validate our perspective. Our results establish that selective underfitting is essential for understanding diffusion models, yielding new, testable insights into their generalization and generative performance.

Related papers

• Deeper Diffusion Models Amplify Bias [54.8794775172033]
  This paper focuses on exploring the notion of bias-variance tradeoff in diffusion models.<n>It establishes that at one extreme, the diffusion models may amplify the inherent bias in the training data, and on the other, they may compromise the presumed privacy of the training samples.
  arXiv  Detail & Related papers  (2025-05-23T07:08:09Z)
• Generalization through variance: how noise shapes inductive biases in diffusion models [0.0]
  We develop a mathematical theory that partly explains 'generalization through variance' phenomenon.<n>We find that the distributions diffusion models effectively learn to sample from resemble their training distributions.<n>We also characterize how this inductive bias interacts with feature-related inductive biases.
  arXiv  Detail & Related papers  (2025-04-16T23:41:10Z)
• Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models [22.39558434131574]
  Existing data attribution methods for diffusion models typically quantify the contribution of a training sample.<n>We argue that the direct usage of diffusion loss cannot represent such a contribution accurately due to the calculation of diffusion loss.<n>We propose Diffusion Attribution Score (textitDAS) to measure the direct comparison between predicted distributions with an attribution score.
  arXiv  Detail & Related papers  (2024-10-24T10:58:17Z)
• Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation [53.27596811146316]
  Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts. We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
  arXiv  Detail & Related papers  (2024-01-17T07:58:18Z)
• Guided Diffusion from Self-Supervised Diffusion Features [49.78673164423208]
  Guidance serves as a key concept in diffusion models, yet its effectiveness is often limited by the need for extra data annotation or pretraining. We propose a framework to extract guidance from, and specifically for, diffusion models.
  arXiv  Detail & Related papers  (2023-12-14T11:19:11Z)
• On the Generalization of Diffusion Model [42.447639515467934]
  We define the generalization of the generative model, which is measured by the mutual information between the generated data and the training set. We show that for the empirical optimal diffusion model, the data generated by a deterministic sampler are all highly related to the training set, thus poor generalization. We propose another training objective whose empirical optimal solution has no potential generalization problem.
  arXiv  Detail & Related papers  (2023-05-24T04:27:57Z)
• A Survey on Generative Diffusion Model [75.93774014861978]
  Diffusion models are an emerging class of deep generative models. They have certain limitations, including a time-consuming iterative generation process and confinement to high-dimensional Euclidean space. This survey presents a plethora of advanced techniques aimed at enhancing diffusion models.
  arXiv  Detail & Related papers  (2022-09-06T16:56:21Z)
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models [76.76860707897413]
  Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution. We show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process.
  arXiv  Detail & Related papers  (2022-06-10T15:09:46Z)
• Right for the Right Latent Factors: Debiasing Generative Models via Disentanglement [20.41752850243945]
  Key assumption of most statistical machine learning methods is that they have access to independent samples from the distribution of data they encounter at test time. In particular, machine learning models have been shown to exhibit Clever-Hans-like behaviour, meaning that spurious correlations in the training set are inadvertently learnt. We propose to debias generative models by disentangling their internal representations, which is achieved via human feedback.
  arXiv  Detail & Related papers  (2022-02-01T13:16:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.