Exploring the Energy Landscape of RBMs: Reciprocal Space Insights into Bosons, Hierarchical Learning and Symmetry Breaking

• URL: http://arxiv.org/abs/2503.21536v1
• Date: Thu, 27 Mar 2025 14:28:37 GMT
• Title: Exploring the Energy Landscape of RBMs: Reciprocal Space Insights into Bosons, Hierarchical Learning and Symmetry Breaking
• Authors: J. Quetzalcóatl Toledo-Marin, Anindita Maiti, Geoffrey C. Fox, Roger G. Melko,
• Abstract summary: We focus on Restricted Boltzmann Machines (RBMs), known for their universal approximation capabilities for discrete distributions.<n>By introducing a reciprocal space formulation, we reveal a connection between RBMs, diffusion processes, and coupled Bosons.<n>Our findings bridge the gap between disparate generative frameworks and also shed light on the processes underpinning learning in generative models.
• Score: 0.07499722271664146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep generative models have become ubiquitous due to their ability to learn and sample from complex distributions. Despite the proliferation of various frameworks, the relationships among these models remain largely unexplored, a gap that hinders the development of a unified theory of AI learning. We address two central challenges: clarifying the connections between different deep generative models and deepening our understanding of their learning mechanisms. We focus on Restricted Boltzmann Machines (RBMs), known for their universal approximation capabilities for discrete distributions. By introducing a reciprocal space formulation, we reveal a connection between RBMs, diffusion processes, and coupled Bosons. We show that at initialization, the RBM operates at a saddle point, where the local curvature is determined by the singular values, whose distribution follows the Marcenko-Pastur law and exhibits rotational symmetry. During training, this rotational symmetry is broken due to hierarchical learning, where different degrees of freedom progressively capture features at multiple levels of abstraction. This leads to a symmetry breaking in the energy landscape, reminiscent of Landau theory. This symmetry breaking in the energy landscape is characterized by the singular values and the weight matrix eigenvector matrix. We derive the corresponding free energy in a mean-field approximation. We show that in the limit of infinite size RBM, the reciprocal variables are Gaussian distributed. Our findings indicate that in this regime, there will be some modes for which the diffusion process will not converge to the Boltzmann distribution. To illustrate our results, we trained replicas of RBMs with different hidden layer sizes using the MNIST dataset. Our findings bridge the gap between disparate generative frameworks and also shed light on the processes underpinning learning in generative models.

Related papers

• Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space [72.52365911990935]
  We introduce Bellman Diffusion, a novel DGM framework that maintains linearity in MDPs through gradient and scalar field modeling. Our results show that Bellman Diffusion achieves accurate field estimations and is a capable image generator, converging 1.5x faster than the traditional histogram-based baseline in distributional RL tasks.
  arXiv  Detail & Related papers  (2024-10-02T17:53:23Z)
• Generative Fractional Diffusion Models [53.36835573822926]
  We introduce the first continuous-time score-based generative model that leverages fractional diffusion processes for its underlying dynamics. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID.
  arXiv  Detail & Related papers  (2023-10-26T17:53:24Z)
• Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processes [57.396578974401734]
  We introduce a principled framework for building a generative diffusion process on general manifold. Instead of following the denoising approach of previous diffusion models, we construct a diffusion process using a mixture of bridge processes. We develop a geometric understanding of the mixture process, deriving the drift as a weighted mean of tangent directions to the data points.
  arXiv  Detail & Related papers  (2023-10-11T06:04:40Z)
• Mirror Diffusion Models for Constrained and Watermarked Generation [41.27274841596343]
  Mirror Diffusion Models (MDM) is a new class of diffusion models that generate data on convex constrained sets without losing tractability. For safety and privacy purposes, we also explore constrained sets as a new mechanism to embed invisible but quantitative information in generated data. Our work brings new algorithmic opportunities for learning tractable diffusion on complex domains.
  arXiv  Detail & Related papers  (2023-10-02T14:26:31Z)
• Lipschitz Singularities in Diffusion Models [64.28196620345808]
  Diffusion models often display the infinite Lipschitz property of the network with respect to time variable near the zero point.<n>We propose a novel approach, dubbed E-TSDM, which alleviates the Lipschitz singularities of the diffusion model near the zero point.<n>Our work may advance the understanding of the general diffusion process, and also provide insights for the design of diffusion models.
  arXiv  Detail & Related papers  (2023-06-20T03:05:28Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• COMET Flows: Towards Generative Modeling of Multivariate Extremes and Tail Dependence [13.041607703862724]
  COMET Flows decompose the process of modeling a joint distribution into two parts: (i) modeling its marginal distributions, and (ii) modeling its copula distribution. Results on both synthetic and real-world datasets demonstrate the effectiveness of COMET Flows.
  arXiv  Detail & Related papers  (2022-05-02T21:37:54Z)
• Generative Ensemble Regression: Learning Particle Dynamics from Observations of Ensembles with Physics-Informed Deep Generative Models [27.623119767592385]
  We propose a new method for inferring the governing ordinary differential equations (SODEs) by observing particle ensembles at discrete and sparse time instants. Particle coordinates at a single time instant, possibly noisy or truncated, are recorded in each snapshot but are unpaired across the snapshots. By training a physics-informed generative model that generates "fake" sample paths, we aim to fit the observed particle ensemble distributions with a curve in the probability measure space.
  arXiv  Detail & Related papers  (2020-08-05T03:06:40Z)
• Targeted free energy estimation via learned mappings [66.20146549150475]
  Free energy perturbation (FEP) was proposed by Zwanzig more than six decades ago as a method to estimate free energy differences. FEP suffers from a severe limitation: the requirement of sufficient overlap between distributions. One strategy to mitigate this problem, called Targeted Free Energy Perturbation, uses a high-dimensional mapping in configuration space to increase overlap.
  arXiv  Detail & Related papers  (2020-02-12T11:10:00Z)
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space [0.0]
  We study the learning dynamics of Restricted Boltzmann Machines (RBM), a neural network paradigm for representation learning. As learning proceeds from a random configuration of the network weights, we show the existence of a symmetry-breaking phenomenon. This symmetry-breaking phenomenon takes place only if the amount of data available for training exceeds some critical value.
  arXiv  Detail & Related papers  (2019-12-30T14:37:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.