The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

• URL: http://arxiv.org/abs/2310.17467v4
• Date: Thu, 20 Jun 2024 08:39:11 GMT
• Title: The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability
• Authors: Luca Ambrogioni,
• Abstract summary: We show that generative diffusion models undergo second-order phase transitions corresponding to universality breaking phenomena. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities.
• Score: 6.4322891559626125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative diffusion models have achieved spectacular performance in many areas of machine learning and generative modeling. While the fundamental ideas behind these models come from non-equilibrium physics, variational inference and stochastic calculus, in this paper we show that many aspects of these models can be understood using the tools of equilibrium statistical mechanics. Using this reformulation, we show that generative diffusion models undergo second-order phase transitions corresponding to symmetry breaking phenomena. We show that these phase-transitions are always in a mean-field universality class, as they are the result of a self-consistency condition in the generative dynamics. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities, which are characterized by a set of mean-field critical exponents. Finally, we show that the dynamic equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in thermal equilibrium.

Related papers

• Nonequilbrium physics of generative diffusion models [2.5690340428649328]
  We provide a transparent physics analysis of the diffusion models, deriving the fluctuation theorem, entropy production, Franz-Parisi potential to understand intrinsic phase transitions. Our analysis is rooted in non-equlibrium physics and concepts from equilibrium physics, i.e., treating both forward and backward dynamics as a Langevin dynamics. This unified principle is expected to guide machine learning practitioners to design better algorithms and theoretical physicists to link the machine learning to non-equilibrium thermodynamics.
  arXiv  Detail & Related papers  (2024-05-20T10:16:26Z)
• Probing non-equilibrium dissipative phase transitions with trapped-ion quantum simulators [0.5356944479760104]
  Open quantum many-body systems with controllable dissipation can exhibit novel features in their dynamics and steady states. We show that strong signatures of this dissipative phase transition and its non-equilibrium properties can be observed with a small system size. Dissipation engineered in this way may allow the simulation of more general types of driven-dissipative systems.
  arXiv  Detail & Related papers  (2023-11-10T17:31:00Z)
• Causal Modeling with Stationary Diffusions [89.94899196106223]
  We learn differential equations whose stationary densities model a system's behavior under interventions. We show that they generalize to unseen interventions on their variables, often better than classical approaches. Our inference method is based on a new theoretical result that expresses a stationarity condition on the diffusion's generator in a reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2023-10-26T14:01:17Z)
• Emergence of a quasi-ergodic steady state in a dissipative Tavis-Cummings array [0.0]
  We show the emergence of a quasi-steady state in a dissipative environment that exhibits intriguing ergodic behavior. The phase space dynamics reveals a fascinating ergodic behavior in presence of dissipation. We discuss the relevance of our findings in the current experiments.
  arXiv  Detail & Related papers  (2023-10-19T14:30:21Z)
• Quantum Effects on the Synchronization Dynamics of the Kuramoto Model [62.997667081978825]
  We show that quantum fluctuations hinder the emergence of synchronization, albeit not entirely suppressing it. We derive an analytical expression for the critical coupling, highlighting its dependence on the model parameters.
  arXiv  Detail & Related papers  (2023-06-16T16:41:16Z)
• Spontaneous Symmetry Breaking in Generative Diffusion Models [6.4322891559626125]
  Generative diffusion models have recently emerged as a leading approach for generating high-dimensional data. We show that the dynamics of these models exhibit a spontaneous symmetry breaking that divides the generative dynamics into two distinct phases. We propose a new way to understand the generative dynamics of diffusion models that has the potential to bring about higher performance and less biased fast-samplers.
  arXiv  Detail & Related papers  (2023-05-31T09:36:34Z)
• Signatures of a quantum phase transition on a single-mode bosonic model [0.0]
  Equilibrium phase transitions emerge from the microscopic behavior of many-body systems. They can be defined through the non-analytic behavior of thermodynamic potentials in the thermodynamic limit. Taking previous ideas to the extreme, we argue that such a limit can be defined even in non-extended systems.
  arXiv  Detail & Related papers  (2023-03-22T20:14:45Z)
• Diffusion Models in Vision: A Survey [80.82832715884597]
  A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens.
  arXiv  Detail & Related papers  (2022-09-10T22:00:30Z)
• GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation [102.85440102147267]
  We propose a novel generative model named GeoDiff for molecular conformation prediction. We show that GeoDiff is superior or comparable to existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-06T09:47:01Z)
• Geometric phase in a dissipative Jaynes-Cummings model: theoretical explanation for resonance robustness [68.8204255655161]
  We compute the geometric phases acquired in both unitary and dissipative Jaynes-Cummings models. In the dissipative model, the non-unitary effects arise from the outflow of photons through the cavity walls. We show the geometric phase is robust, exhibiting a vanishing correction under a non-unitary evolution.
  arXiv  Detail & Related papers  (2021-10-27T15:27:54Z)
• Analog cosmological reheating in an ultracold Bose gas [58.720142291102135]
  We quantum-simulate the reheating-like dynamics of a generic cosmological single-field model in an ultracold Bose gas. Expanding spacetime as well as the background oscillating inflaton field are mimicked in the non-relativistic limit. The proposed experiment has the potential of exploring the evolution up to late times even beyond the weak coupling regime.
  arXiv  Detail & Related papers  (2020-08-05T18:00:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.