Conservation Laws for Diffusion Models

• URL: http://arxiv.org/abs/2607.10067v1
• Date: Sat, 11 Jul 2026 01:37:52 GMT
• Title: Conservation Laws for Diffusion Models
• Abstract summary: We develop conservation laws for a class of memoryless noise processes.<n>We show that the data--model cross-entropy (CE) can be characterized exactly as an integral of local information-theoretic derivatives along the noise path.<n>We validate these predictions on synthetic Markov sources and standard benchmarks.
• Score: 10.98617584718584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While autoregressive models optimize the exact data likelihood via the chain rule, diffusion models are typically trained with denoising objectives. We develop conservation laws based on generalized extrinsic information transfer (GEXIT) functions for a broad class of memoryless noise processes, showing that the data--model cross-entropy (CE) can be characterized exactly as an integral of local information-theoretic derivatives along the noise path. This yields a unified characterization of the likelihood for discrete and continuous diffusion, with the Gaussian case reducing to the well-known mutual information--minimum mean-square error (I-MMSE) relationship. An immediate implication is a locality property: one can compute the information-theoretic derivatives using only the marginal posteriors along the noise path. As a result, training reduces to learning the marginal posteriors by minimizing the negative log-likelihood. While the conservation law implies that the entropy does not depend on the noise path, finite-capacity denoisers approximate the posteriors with varying accuracy across noise types, leading to differences in performance. We validate these predictions on synthetic Markov sources and standard benchmarks, including text8 and CIFAR-10.

Related papers

• Posterior Information Dynamics of Diffusion Models for Linear Inverse Problems [11.548853370822343]
  We study the process through the smoothed likelihood force, the difference between exact posterior and prior scores at each noise level.<n>Averaging over measurements yields an information--minimum mean-square error (I-MMSE) identity linking information gain to denoising-error reduction.
  arXiv  Detail & Related papers  (2026-08-22T00:53:42Z)
• UniNDM: A Unified Noise-driven Detection and Mitigation Framework Against Sexual Content in Text-to-Image Generation [90.0896070825457]
  We propose a unified noise-driven framework that rethinks safety mechanisms through the lens of noise dynamics in diffusion processes.<n>Our key insight is that early-stage predicted noise exhibits inherent separability between normal and sexually explicit content.<n>We develop a lightweight noise-based detector achieving superior accuracy with virtually no computational overhead.
  arXiv  Detail & Related papers  (2026-07-18T14:03:39Z)
• Nonparametric Deconvolution and Denoising using Simulation Based Inference [3.6640504352010885]
  We learn a latent generative model by matching the observed data distribution to the noise-convolved model distribution.<n>Our method provides a practical, theoretically grounded approach to deconvolution under generative latent distribution models.
  arXiv  Detail & Related papers  (2026-06-20T07:03:37Z)
• Information-Theoretic Discrete Diffusion [8.018632880023336]
  We present an information-theoretic framework for discrete diffusion models that yields principled estimators of log-likelihood using score-matching losses.<n>Results provide a time-integral decomposition of the log-likelihood of the data in terms of optimal score-based losses.<n>Experiments on synthetic and real-world data confirm the accuracy, variance stability, and utility of our estimators.
  arXiv  Detail & Related papers  (2025-10-28T05:59:05Z)
• Information Theoretic Learning for Diffusion Models with Warm Start [8.455757095201314]
  We derive a tighter likelihood bound for noise driven models to improve the accuracy and efficiency of maximum likelihood learning.<n>Our key insight extends the classical KL divergence Fisher information relationship to arbitrary noise perturbations.<n>Treating the diffusion process as a Gaussian channel, we show that the proposed objective upper bounds the negative log-likelihood (NLL)
  arXiv  Detail & Related papers  (2025-10-23T18:00:59Z)
• Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise [43.07594740645669]
  Diffusion Probabilistic Models (DPMs) have achieved strong generative performance, yet their inductive biases remain largely implicit.<n>In this work, we aim to build inductive biases into the training and sampling of diffusion models to better accommodate the target distribution of the data to model.<n>We introduce an anisotropic noise operator that shapes these biases by replacing the isotropic forward covariance with a structured, frequency-diagonal covariance.
  arXiv  Detail & Related papers  (2025-10-07T16:08:39Z)
• Rectified Diffusion Guidance for Conditional Generation [94.83538269086613]
  We revisit the theory behind CFG and rigorously confirm that the improper combination coefficients (textiti.e.) brings about expectation shift the generative distribution.<n>We show that our approach enjoys a textbftextitform solution given the strength.<n> Empirical evidence on real-world data demonstrate the compatibility of our design with existing state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2024-10-24T13:41:32Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• ManiFlow: Implicitly Representing Manifolds with Normalizing Flows [145.9820993054072]
  Normalizing Flows (NFs) are flexible explicit generative models that have been shown to accurately model complex real-world data distributions. We propose an optimization objective that recovers the most likely point on the manifold given a sample from the perturbed distribution. Finally, we focus on 3D point clouds for which we utilize the explicit nature of NFs, i.e. surface normals extracted from the gradient of the log-likelihood and the log-likelihood itself.
  arXiv  Detail & Related papers  (2022-08-18T16:07:59Z)
• Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD [73.55632827932101]
  We optimize the information-theoretical generalization bound by manipulating the noise structure in SGLD. We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance.
  arXiv  Detail & Related papers  (2021-10-26T15:02:27Z)
• Discrete Denoising Flows [87.44537620217673]
  We introduce a new discrete flow-based model for categorical random variables: Discrete Denoising Flows (DDFs) In contrast with other discrete flow-based models, our model can be locally trained without introducing gradient bias. We show that DDFs outperform Discrete Flows on modeling a toy example, binary MNIST and Cityscapes segmentation maps, measured in log-likelihood.
  arXiv  Detail & Related papers  (2021-07-24T14:47:22Z)
• Adaptive Multi-View ICA: Estimation of noise levels for optimal inference [65.94843987207445]
  Adaptive multiView ICA (AVICA) is a noisy ICA model where each view is a linear mixture of shared independent sources with additive noise on the sources. On synthetic data, AVICA yields better sources estimates than other group ICA methods thanks to its explicit MMSE estimator. On real magnetoencephalograpy (MEG) data, we provide evidence that the decomposition is less sensitive to sampling noise and that the noise variance estimates are biologically plausible.
  arXiv  Detail & Related papers  (2021-02-22T13:10:12Z)
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance [76.54300276636982]
  Noise in gradient descent provides a crucial implicit regularization effect for training over parameterized models. We show that parameter-dependent noise -- induced by mini-batches or label perturbation -- is far more effective than Gaussian noise. Our analysis reveals that parameter-dependent noise introduces a bias towards local minima with smaller noise variance, whereas spherical Gaussian noise does not.
  arXiv  Detail & Related papers  (2020-06-15T18:31:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.