From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime

• URL: http://arxiv.org/abs/2607.20540v1
• Date: Fri, 10 Jul 2026 20:59:06 GMT
• Title: From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime
• Abstract summary: We develop a general statistical framework for studying optimal noise-level allocation in diffusion training.<n>We show that the optimized training schedule admits an atomic minimizer, concentrated on finitely many noise levels.<n>We then evaluate the entropic schedule in larger-scale experiments, where full schedule optimization is currently intractable.
• Score: 52.459513910242485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How should a diffusion model decide which noise levels to train on, and how much? Despite the importance of this choice, current noise schedules are based largely on heuristics or empirical tuning. Here, we develop a general statistical framework for studying asymptotically optimal noise-level allocation in diffusion training. Our first main result concerns the fully coupled regime, where information can spread between different time points. Under convexity or Polyak-Lojasiewicz-type assumptions, we show that the optimized training schedule admits an atomic minimizer, concentrated on finitely many noise levels. Our second main result specializes this framework to an idealized independent-learner regime, intended to model temporal specialization in neural networks. Under an additional feature-noise decoupling condition, a random-matrix analysis leads to an information-theoretic proxy: the decoupled sampling density is proportional to the square root of the generative entropy rate, the rate at which conditional entropy grows along the forward process. We test these predictions in controlled settings where the coupled objective can be optimized directly, including Dirac mixtures, low-dimensional manifolds, and MNIST. In these settings, the optimized schedules are consistently finite-support, while the smooth entropic proxy closely tracks the atomic optimum in neural-network models and breaks down mainly in the fully coupled parametric case, as the theory suggests. We then evaluate the entropic schedule in larger-scale experiments, where full schedule optimization is currently intractable. The results indicate that square-root entropy scheduling can substantially improve training efficiency on discrete domains and remains competitive with standard EDM-style heuristics on continuous images.

Related papers

• A Defense of the Quadratic Model [71.57026347416729]
  We stress test the simplest possible model of optimization -- the quadratic model.<n>We show that it can be predictive surprisingly in an LLM setting with 150M parameters and 3B training tokens.
  arXiv  Detail & Related papers  (2026-07-23T18:02:03Z)
• Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning [16.158545640309438]
  We develop a statistical inference framework for gradient-flow training itself.<n>Training time becomes a variance-component parameter governing how variance is reallocated.<n>Deep learning models in fixed gradient regimes provide modern-AI instantiations of the theory.
  arXiv  Detail & Related papers  (2026-05-27T05:32:24Z)
• Information-Guided Noise Allocation for Efficient Diffusion Training [50.49440376762109]
  InfoNoise is a principled data-adaptive training noise schedule that replaces schedule design with an information-guided noise sampling distribution.<n>Across natural-image benchmarks, InfoNoise matches or surpasses tuned EDM-style schedules, in some cases with a substantial training speedup.<n>Overall, InfoNoise makes noise scheduling data-adaptive, reducing the need for per-dataset schedule design as diffusion models expand across domains.
  arXiv  Detail & Related papers  (2026-02-20T22:45:01Z)
• Low-Dimensional Adaptation of Rectified Flow: A New Perspective through the Lens of Diffusion and Stochastic Localization [59.04314685837778]
  Rectified flow (RF) has gained considerable popularity due to its generation efficiency and state-of-the-art performance.<n>In this paper, we investigate the degree to which RF automatically adapts to the intrinsic low dimensionality of the support of the target distribution to accelerate sampling.<n>We show that, using a carefully designed choice of the time-discretization scheme and with sufficiently accurate drift estimates, the RF sampler enjoys an complexity of order $O(k/varepsilon)$.
  arXiv  Detail & Related papers  (2026-01-21T22:09:27Z)
• Matching the Optimal Denoiser in Point Cloud Diffusion with (Improved) Rotational Alignment [5.8069334875117775]
  Training a diffusion model consists of learning how to denoise noisy samples at different noise levels.<n>We show that the optimal denoiser can be expressed in terms of a matrix Fisher distribution over $SO(3)$.<n>We build on this perspective to derive better approximators to the optimal denoiser in the limit of small noise.
  arXiv  Detail & Related papers  (2025-10-02T05:55:22Z)
• Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data [12.815892583089443]
  This paper introduces an architecture-specific randomized training algorithm that constructs a uniform approximator from $N$ noisy training samples.<n>Our trained neural networks attain the minimax-optimal quantity of textittrainable (non-random) parameters, subject to logarithmic factors which vanish under the idealized noiseless sampling assumed in classical UATs.
  arXiv  Detail & Related papers  (2025-08-31T16:20:27Z)
• Generative Latent Neural PDE Solver using Flow Matching [8.397730500554047]
  We propose a latent diffusion model for PDE simulation that embeds the PDE state in a lower-dimensional latent space.<n>Our framework uses an autoencoder to map different types of meshes onto a unified structured latent grid, capturing complex geometries.<n> Numerical experiments show that the proposed model outperforms several deterministic baselines in both accuracy and long-term stability.
  arXiv  Detail & Related papers  (2025-03-28T16:44:28Z)
• Accelerated zero-order SGD under high-order smoothness and overparameterized regime [79.85163929026146]
  We present a novel gradient-free algorithm to solve convex optimization problems. Such problems are encountered in medicine, physics, and machine learning. We provide convergence guarantees for the proposed algorithm under both types of noise.
  arXiv  Detail & Related papers  (2024-11-21T10:26:17Z)
• Improved Noise Schedule for Diffusion Training [51.849746576387375]
  We propose a novel approach to design the noise schedule for enhancing the training of diffusion models.<n>We empirically demonstrate the superiority of our noise schedule over the standard cosine schedule.
  arXiv  Detail & Related papers  (2024-07-03T17:34:55Z)
• Learning Unnormalized Statistical Models via Compositional Optimization [73.30514599338407]
  Noise-contrastive estimation(NCE) has been proposed by formulating the objective as the logistic loss of the real data and the artificial noise. In this paper, we study it a direct approach for optimizing the negative log-likelihood of unnormalized models.
  arXiv  Detail & Related papers  (2023-06-13T01:18:16Z)
• Self-Adapting Noise-Contrastive Estimation for Energy-Based Models [0.0]
  Training energy-based models with noise-contrastive estimation (NCE) is theoretically feasible but practically challenging. Previous works have explored modelling the noise distribution as a separate generative model, and then concurrently training this noise model with the EBM. This thesis proposes a self-adapting NCE algorithm which uses static instances of the EBM along its training trajectory as the noise distribution.
  arXiv  Detail & Related papers  (2022-11-03T15:17:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.