The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

• URL: http://arxiv.org/abs/2310.17467v4
• Date: Thu, 20 Jun 2024 08:39:11 GMT
• Title: The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability
• Authors: Luca Ambrogioni,
• Abstract summary: We show that generative diffusion models undergo second-order phase transitions corresponding to universality breaking phenomena. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities.
• Score: 6.4322891559626125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative diffusion models have achieved spectacular performance in many areas of machine learning and generative modeling. While the fundamental ideas behind these models come from non-equilibrium physics, variational inference and stochastic calculus, in this paper we show that many aspects of these models can be understood using the tools of equilibrium statistical mechanics. Using this reformulation, we show that generative diffusion models undergo second-order phase transitions corresponding to symmetry breaking phenomena. We show that these phase-transitions are always in a mean-field universality class, as they are the result of a self-consistency condition in the generative dynamics. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities, which are characterized by a set of mean-field critical exponents. Finally, we show that the dynamic equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in thermal equilibrium.

Related papers

• Energy-Based Diffusion Language Models for Text Generation [126.23425882687195]
  Energy-based Diffusion Language Model (EDLM) is an energy-based model operating at the full sequence level for each diffusion step. Our framework offers a 1.3$times$ sampling speedup over existing diffusion models.
  arXiv  Detail & Related papers  (2024-10-28T17:25:56Z)
• Geometric Trajectory Diffusion Models [58.853975433383326]
  Generative models have shown great promise in generating 3D geometric systems. Existing approaches only operate on static structures, neglecting the fact that physical systems are always dynamic in nature. We propose geometric trajectory diffusion models (GeoTDM), the first diffusion model for modeling the temporal distribution of 3D geometric trajectories.
  arXiv  Detail & Related papers  (2024-10-16T20:36:41Z)
• Nonequilbrium physics of generative diffusion models [2.5690340428649328]
  Generative diffusion models apply the concept of Langevin dynamics in physics to machine leaning. We provide a transparent physics analysis of diffusion models, formulating the fluctuation theorem, entropy production, equilibrium measure, and Franz-Parisi potential. Our study links thermodynamics, statistical inference and geometry based analysis together to yield a coherent picture about how the generative diffusion models work.
  arXiv  Detail & Related papers  (2024-05-20T10:16:26Z)
• Probing non-equilibrium dissipative phase transitions with trapped-ion quantum simulators [0.5356944479760104]
  Open quantum many-body systems with controllable dissipation can exhibit novel features in their dynamics and steady states. We show that strong signatures of this dissipative phase transition and its non-equilibrium properties can be observed with a small system size. Dissipation engineered in this way may allow the simulation of more general types of driven-dissipative systems.
  arXiv  Detail & Related papers  (2023-11-10T17:31:00Z)
• Causal Modeling with Stationary Diffusions [89.94899196106223]
  We learn differential equations whose stationary densities model a system's behavior under interventions. We show that they generalize to unseen interventions on their variables, often better than classical approaches. Our inference method is based on a new theoretical result that expresses a stationarity condition on the diffusion's generator in a reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2023-10-26T14:01:17Z)
• Emergence of a quasi-ergodic steady state in a dissipative Tavis-Cummings array [0.0]
  We show the emergence of a quasi-steady state in a dissipative environment that exhibits intriguing ergodic behavior. The phase space dynamics reveals a fascinating ergodic behavior in presence of dissipation. We discuss the relevance of our findings in the current experiments.
  arXiv  Detail & Related papers  (2023-10-19T14:30:21Z)
• Quantum Effects on the Synchronization Dynamics of the Kuramoto Model [62.997667081978825]
  We show that quantum fluctuations hinder the emergence of synchronization, albeit not entirely suppressing it. We derive an analytical expression for the critical coupling, highlighting its dependence on the model parameters.
  arXiv  Detail & Related papers  (2023-06-16T16:41:16Z)
• Spontaneous Symmetry Breaking in Generative Diffusion Models [6.4322891559626125]
  Generative diffusion models have recently emerged as a leading approach for generating high-dimensional data. We show that the dynamics of these models exhibit a spontaneous symmetry breaking that divides the generative dynamics into two distinct phases. We propose a new way to understand the generative dynamics of diffusion models that has the potential to bring about higher performance and less biased fast-samplers.
  arXiv  Detail & Related papers  (2023-05-31T09:36:34Z)
• Signatures of a quantum phase transition on a single-mode bosonic model [0.0]
  Equilibrium phase transitions emerge from the microscopic behavior of many-body systems. They can be defined through the non-analytic behavior of thermodynamic potentials in the thermodynamic limit. Taking previous ideas to the extreme, we argue that such a limit can be defined even in non-extended systems.
  arXiv  Detail & Related papers  (2023-03-22T20:14:45Z)
• GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation [102.85440102147267]
  We propose a novel generative model named GeoDiff for molecular conformation prediction. We show that GeoDiff is superior or comparable to existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-06T09:47:01Z)
• Geometric phase in a dissipative Jaynes-Cummings model: theoretical explanation for resonance robustness [68.8204255655161]
  We compute the geometric phases acquired in both unitary and dissipative Jaynes-Cummings models. In the dissipative model, the non-unitary effects arise from the outflow of photons through the cavity walls. We show the geometric phase is robust, exhibiting a vanishing correction under a non-unitary evolution.
  arXiv  Detail & Related papers  (2021-10-27T15:27:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.