Measuring Semantic Information Production in Generative Diffusion Models

• URL: http://arxiv.org/abs/2506.10433v1
• Date: Thu, 12 Jun 2025 07:35:29 GMT
• Title: Measuring Semantic Information Production in Generative Diffusion Models
• Authors: Florian Handke, Félix Koulischer, Gabriel Raya, Luca Ambrogioni,
• Abstract summary: It is well known that semantic and structural features emerge at different times during the reverse dynamics of diffusion.<n>We introduce a general information-theoretic approach to measure when these class-semantic "decisions" are made during the generative process.
• Score: 4.133545163830844
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It is well known that semantic and structural features of the generated images emerge at different times during the reverse dynamics of diffusion, a phenomenon that has been connected to physical phase transitions in magnets and other materials. In this paper, we introduce a general information-theoretic approach to measure when these class-semantic "decisions" are made during the generative process. By using an online formula for the optimal Bayesian classifier, we estimate the conditional entropy of the class label given the noisy state. We then determine the time intervals corresponding to the highest information transfer between noisy states and class labels using the time derivative of the conditional entropy. We demonstrate our method on one-dimensional Gaussian mixture models and on DDPM models trained on the CIFAR10 dataset. As expected, we find that the semantic information transfer is highest in the intermediate stages of diffusion while vanishing during the final stages. However, we found sizable differences between the entropy rate profiles of different classes, suggesting that different "semantic decisions" are located at different intermediate times.

Related papers

• The Entropic Signature of Class Speciation in Diffusion Models [12.212582407125089]
  We show that tracking the class-conditional entropy of a latent semantic variable given a noisy state provides a reliable signature of transition regimes.<n>We validate our method on EDM2-XS and Stable Diffusion 1.5, where class-conditional entropy consistently isolates the noise regimes critical for semantic structure formation.
  arXiv  Detail & Related papers  (2026-02-10T10:56:46Z)
• Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold [19.059115911590776]
  We explicitly characterize what diffusion model generates, by proposing a log-density ridge manifold.<n>We show how the generated data relate to this manifold as inference dynamics progresses.<n>More detailed understanding of training dynamics will lead to more accurate quantification of the generation inductive bias.
  arXiv  Detail & Related papers  (2026-02-05T18:55:03Z)
• Theory of Speciation Transitions in Diffusion Models with General Class Structure [5.939780039158003]
  Diffusion models generate data by reversing a diffusion process, transforming noise into structured samples drawn from a target distribution.<n>Recent theoretical work has shown that this backward dynamics can undergo sharp qualitative transitions, known as speciation transitions.<n>We develop a general theory of speciation in diffusion models that applies to arbitrary target distributions admitting well-defined classes.
  arXiv  Detail & Related papers  (2026-02-04T10:35:47Z)
• Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction [54.95522167029998]
  This article is a self-contained primer on diffusion over general state spaces.<n>We develop the discrete-time view (forward noising via Markov kernels and learned reverse dynamics) alongside its continuous-time limits.<n>A common variational treatment yields the ELBO that underpins standard training losses.
  arXiv  Detail & Related papers  (2025-12-04T18:55:36Z)
• The Principles of Diffusion Models [81.12042238390075]
  Diffusion modeling starts by defining a forward process that gradually corrupts data into noise.<n>The goal is to learn a reverse process that transforms noise back into data while recovering the same intermediates.<n>The score-based view, rooted in energy-based modeling, learns the gradient of the evolving data distribution.<n>The flow-based view, related to normalizing flows, treats generation as following a smooth path that moves samples from noise to data.
  arXiv  Detail & Related papers  (2025-10-24T02:29:02Z)
• Authentic Discrete Diffusion Model [72.31371542619121]
  Authentic Discrete Diffusion (ADD) framework redefines prior pseudo-discrete approaches.<n>ADD reformulates the diffusion input by directly using float-encoded one-hot class data.<n> experiments demonstrate that ADD achieves superior performance on classification tasks compared to the baseline.
  arXiv  Detail & Related papers  (2025-10-01T15:51:10Z)
• Diffusion models for multivariate subsurface generation and efficient probabilistic inversion [0.0]
  Diffusion models offer stable training and state-of-the-art performance for deep generative modeling tasks.<n>We introduce a likelihood approximation accounting for the noise-contamination that is inherent in diffusion modeling.<n>Our tests show significantly improved statistical robustness, enhanced sampling of the posterior probability density function.
  arXiv  Detail & Related papers  (2025-07-21T17:10:16Z)
• Conditional Synthesis of 3D Molecules with Time Correction Sampler [58.0834973489875]
  Time-Aware Conditional Synthesis (TACS) is a novel approach to conditional generation on diffusion models. It integrates adaptively controlled plug-and-play "online" guidance into a diffusion model, driving samples toward the desired properties.
  arXiv  Detail & Related papers  (2024-11-01T12:59:25Z)
• Meta Flow Matching: Integrating Vector Fields on the Wasserstein Manifold [83.18058549195855]
  We argue that multiple processes in natural sciences have to be represented as vector fields on the Wasserstein manifold of probability densities.<n>In particular, this is crucial for personalized medicine where the development of diseases and their respective treatment response depend on the microenvironment of cells specific to each patient.<n>We propose Meta Flow Matching (MFM), a practical approach to integrate along these vector fields on the Wasserstein manifold by amortizing the flow model over the initial populations.
  arXiv  Detail & Related papers  (2024-08-26T20:05:31Z)
• Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models [59.331993845831946]
  Diffusion models benefit from instillation of task-specific information into the score function to steer the sample generation towards desired properties. This paper provides the first theoretical study towards understanding the influence of guidance on diffusion models in the context of Gaussian mixture models.
  arXiv  Detail & Related papers  (2024-03-03T23:15:48Z)
• Dynamical Regimes of Diffusion Models [14.797301819675454]
  We study generative diffusion models in the regime where the dimension of space and the number of data are large. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models.
  arXiv  Detail & Related papers  (2024-02-28T17:19:26Z)
• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• Blackout Diffusion: Generative Diffusion Models in Discrete-State Spaces [0.0]
  We develop a theoretical formulation for arbitrary discrete-state Markov processes in the forward diffusion process. As an example, we introduce Blackout Diffusion'', which learns to produce samples from an empty image instead of from noise.
  arXiv  Detail & Related papers  (2023-05-18T16:24:12Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• DiffPhase: Generative Diffusion-based STFT Phase Retrieval [15.16865739526702]
  Diffusion probabilistic models have been recently used in a variety of tasks, including speech enhancement and synthesis. In this work we build upon previous work in the speech domain, adapting a speech enhancement diffusion model specifically for phase retrieval. Evaluation using speech quality and intelligibility metrics shows the diffusion approach is well-suited to the phase retrieval task, with performance surpassing both classical and modern methods.
  arXiv  Detail & Related papers  (2022-11-08T15:50:35Z)
• Time-inhomogeneous diffusion geometry and topology [69.55228523791897]
  Diffusion condensation is a time-inhomogeneous process where each step first computes and then applies a diffusion operator to the data. We theoretically analyze the convergence and evolution of this process from geometric, spectral, and topological perspectives. Our work gives theoretical insights into the convergence of diffusion condensation, and shows that it provides a link between topological and geometric data analysis.
  arXiv  Detail & Related papers  (2022-03-28T16:06:17Z)
• GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation [102.85440102147267]
  We propose a novel generative model named GeoDiff for molecular conformation prediction. We show that GeoDiff is superior or comparable to existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-06T09:47:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.