The Information Dynamics of Generative Diffusion

• URL: http://arxiv.org/abs/2508.19897v3
• Date: Thu, 11 Sep 2025 14:30:28 GMT
• Title: The Information Dynamics of Generative Diffusion
• Authors: Luca Ambrogioni,
• Abstract summary: Generative diffusion models have emerged as a powerful class of models in machine learning.<n>This paper provides an integrated perspective on generative diffusion by connecting their dynamic, information-theoretic, and thermodynamic properties.
• Score: 12.52425103385255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative diffusion models have emerged as a powerful class of models in machine learning, yet a unified theoretical understanding of their operation is still developing. This paper provides an integrated perspective on generative diffusion by connecting their dynamic, information-theoretic, and thermodynamic properties under a unified mathematical framework. We demonstrate that the rate of conditional entropy production during generation (i.e. the generative bandwidth) is directly governed by the expected divergence of the score function's vector field. This divergence, in turn, is linked to the branching of trajectories and generative bifurcations, which we characterize as symmetry-breaking phase transitions in the energy landscape. This synthesis offers a powerful insight: the process of generation is fundamentally driven by the controlled, noise-induced breaking of (approximate) symmetries, where peaks in information transfer correspond to critical transitions between possible outcomes. The score function acts as a dynamic non-linear filter that regulates the bandwidth of the noise by suppressing fluctuations that are incompatible with the data.

Related papers

• Spectral Generative Flow Models: A Physics-Inspired Replacement for Vectorized Large Language Models [0.0]
  We introduce Spectral Generative Flow Models (SGFMs), a physics-inspired alternative to transformer-based large language models.<n>Instead of representing text or video as sequences of discrete tokens processed by attention, SGFMs treat generation as the evolution of a continuous field governed by constrained dynamics.
  arXiv  Detail & Related papers  (2026-01-13T12:50:24Z)
• A Free Probabilistic Framework for Denoising Diffusion Models: Entropy, Transport, and Reverse Processes [22.56299060022639]
  This paper builds on Voiculescu's theory of free entropy and free Fisher information.<n>We formulate diffusion and quantify reverse processes governed by operator-valued dynamics.<n>The resulting dynamics admit a gradient-flow structure in the noncommutative Wasserstein space.
  arXiv  Detail & Related papers  (2025-10-26T18:03:54Z)
• The Principles of Diffusion Models [81.12042238390075]
  Diffusion modeling starts by defining a forward process that gradually corrupts data into noise.<n>The goal is to learn a reverse process that transforms noise back into data while recovering the same intermediates.<n>The score-based view, rooted in energy-based modeling, learns the gradient of the evolving data distribution.<n>The flow-based view, related to normalizing flows, treats generation as following a smooth path that moves samples from noise to data.
  arXiv  Detail & Related papers  (2025-10-24T02:29:02Z)
• Kuramoto Orientation Diffusion Models [67.0711709825854]
  Orientation-rich images, such as fingerprints and textures, often exhibit coherent angular patterns.<n>Motivated by the role of phase synchronization in biological systems, we propose a score-based generative model.<n>We implement competitive results on general image benchmarks and significantly improves generation quality on orientation-dense datasets like fingerprints and textures.
  arXiv  Detail & Related papers  (2025-09-18T18:18:49Z)
• Loss-Complexity Landscape and Model Structure Functions [53.92822954974537]
  We develop a framework for dualizing the Kolmogorov structure function $h_x(alpha)$.<n>We establish a mathematical analogy between information-theoretic constructs and statistical mechanics.<n>We explicitly prove the Legendre-Fenchel duality between the structure function and free energy.
  arXiv  Detail & Related papers  (2025-07-17T21:31:45Z)
• Energy Matching: Unifying Flow Matching and Energy-Based Models for Generative Modeling [4.395339671282145]
  Energy-based models (EBMs) map noise and data distributions by matching flows or scores.<n>We propose Energy Matching, a framework that endows flow-based approaches with the flexibility of EBMs.<n>Our method substantially outperforms existing EBMs on CIFAR-10 and ImageNet generation in terms of fidelity.
  arXiv  Detail & Related papers  (2025-04-14T18:10:58Z)
• Simple and Critical Iterative Denoising: A Recasting of Discrete Diffusion in Graph Generation [0.0]
  dependencies between intermediate noisy states lead to error accumulation and propagation during the reverse denoising process.<n>We propose a novel framework called Simple Iterative Denoising, which simplifies discrete diffusion and circumvents the issue.<n>Our empirical evaluations demonstrate that the proposed method significantly outperforms existing discrete diffusion baselines in graph generation tasks.
  arXiv  Detail & Related papers  (2025-03-27T15:08:58Z)
• Hessian-Informed Flow Matching [4.542719108171107]
  Hessian-Informed Flow Matching is a novel approach that integrates the Hessian of an energy function into conditional flows. This integration allows HI-FM to account for local curvature and anisotropic covariance structures. Empirical evaluations on the MNIST and Lennard-Jones particles datasets demonstrate that HI-FM improves the likelihood of test samples.
  arXiv  Detail & Related papers  (2024-10-15T09:34:52Z)
• Transformers from Diffusion: A Unified Framework for Neural Message Passing [79.9193447649011]
  Message passing neural networks (MPNNs) have become a de facto class of model solutions.<n>We propose an energy-constrained diffusion model, which integrates the inductive bias of diffusion with layer-wise constraints of energy.<n>Building on these insights, we devise a new class of message passing models, dubbed Transformers (DIFFormer), whose global attention layers are derived from the principled energy-constrained diffusion framework.
  arXiv  Detail & Related papers  (2024-09-13T17:54:41Z)
• Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data [39.41800375686212]
  Diffusion Transformer, the backbone of Sora for video generation, successfully scales the capacity of diffusion models.<n>We make the first theoretical step towards bridging diffusion transformers for capturing spatial-temporal dependencies.<n>We highlight how the spatial-temporal dependencies are captured and affect learning efficiency.
  arXiv  Detail & Related papers  (2024-07-23T02:42:43Z)
• Dynamical Regimes of Diffusion Models [14.797301819675454]
  We study generative diffusion models in the regime where the dimension of space and the number of data are large. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models.
  arXiv  Detail & Related papers  (2024-02-28T17:19:26Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Flowformer: Linearizing Transformers with Conservation Flows [77.25101425464773]
  We linearize Transformers free from specific inductive biases based on the flow network theory. By respectively conserving the incoming flow of sinks for source competition and the outgoing flow of sources for sink allocation, Flow-Attention inherently generates informative attentions.
  arXiv  Detail & Related papers  (2022-02-13T08:44:10Z)
• Inference and De-Noising of Non-Gaussian Particle Distribution Functions: A Generative Modeling Approach [0.0]
  Inference on data produced by numerical simulations generally consists of binning the data to recover the particle distribution function. Here we demonstrate the use of normalizing flows to learn a smooth, tractable approximation to the noisy particle distribution function.
  arXiv  Detail & Related papers  (2021-10-05T16:38:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.