'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

• URL: http://arxiv.org/abs/1912.12942v1
• Date: Mon, 30 Dec 2019 14:37:14 GMT
• Title: 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• Authors: Moshir Harsh (LPENS, PSL), J\'er\^ome Tubiana (TAU-CS), Simona Cocco (LPENS, PSL), Remi Monasson (LPENS, PSL)
• Abstract summary: We study the learning dynamics of Restricted Boltzmann Machines (RBM), a neural network paradigm for representation learning. As learning proceeds from a random configuration of the network weights, we show the existence of a symmetry-breaking phenomenon. This symmetry-breaking phenomenon takes place only if the amount of data available for training exceeds some critical value.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distributions of data or sensory stimuli often enjoy underlying invariances. How and to what extent those symmetries are captured by unsupervised learning methods is a relevant question in machine learning and in computational neuroscience. We study here, through a combination of numerical and analytical tools, the learning dynamics of Restricted Boltzmann Machines (RBM), a neural network paradigm for representation learning. As learning proceeds from a random configuration of the network weights, we show the existence of, and characterize a symmetry-breaking phenomenon, in which the latent variables acquire receptive fields focusing on limited parts of the invariant manifold supporting the data. The symmetry is restored at large learning times through the diffusion of the receptive field over the invariant manifold; hence, the RBM effectively spans a continuous attractor in the space of network weights. This symmetry-breaking phenomenon takes place only if the amount of data available for training exceeds some critical value, depending on the network size and the intensity of symmetry-induced correlations in the data; below this 'retarded-learning' threshold, the network weights are essentially noisy and overfit the data.

Related papers

• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• Tipping Points of Evolving Epidemiological Networks: Machine Learning-Assisted, Data-Driven Effective Modeling [0.0]
  We study the tipping point collective dynamics of an adaptive susceptible-infected (SIS) epidemiological network in a data-driven, machine learning-assisted manner. We identify a complex effective differential equation (eSDE) in terms physically meaningful coarse mean-field variables. We study the statistics of rare events both through repeated brute force simulations and by using established mathematical/computational tools.
  arXiv  Detail & Related papers  (2023-11-01T19:33:03Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• Imaginary components of out-of-time correlators and information scrambling for navigating the learning landscape of a quantum machine learning model [0.0]
  We analytically illustrate that hitherto unexplored imaginary components of out-of-time correlators can provide unprecedented insight into the information scrambling capacity of a graph neural network. Such an analysis demystifies the training of quantum machine learning models by unraveling how quantum information is scrambled through such a network.
  arXiv  Detail & Related papers  (2022-08-29T06:17:28Z)
• Analyzing the Effects of Handling Data Imbalance on Learned Features from Medical Images by Looking Into the Models [50.537859423741644]
  Training a model on an imbalanced dataset can introduce unique challenges to the learning problem. We look deeper into the internal units of neural networks to observe how handling data imbalance affects the learned features.
  arXiv  Detail & Related papers  (2022-04-04T09:38:38Z)
• Learning Invariant Weights in Neural Networks [16.127299898156203]
  Many commonly used models in machine learning are constraint to respect certain symmetries in the data. We propose a weight-space equivalent to this approach, by minimizing a lower bound on the marginal likelihood to learn invariances in neural networks.
  arXiv  Detail & Related papers  (2022-02-25T00:17:09Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Symplectic Momentum Neural Networks -- Using Discrete Variational Mechanics as a prior in Deep Learning [7.090165638014331]
  This paper introduces Sympic Momentum Networks (SyMo) as models from a discrete formulation of mechanics for non-separable mechanical systems. We show that such combination not only allows these models tol earn from limited data but also provides the models with the capability of preserving the symplectic form and show better long-term behaviour.
  arXiv  Detail & Related papers  (2022-01-20T16:33:19Z)
• Learning Invariances in Neural Networks [51.20867785006147]
  We show how to parameterize a distribution over augmentations and optimize the training loss simultaneously with respect to the network parameters and augmentation parameters. We can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations.
  arXiv  Detail & Related papers  (2020-10-22T17:18:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.