Renormalization in the neural network-quantum field theory correspondence

• URL: http://arxiv.org/abs/2212.11811v1
• Date: Thu, 22 Dec 2022 15:41:13 GMT
• Title: Renormalization in the neural network-quantum field theory correspondence
• Authors: Harold Erbin, Vincent Lahoche, Dine Ousmane Samary
• Abstract summary: A statistical ensemble of neural networks can be described in terms of a quantum field theory. A major outcome is that changing the standard deviation of the neural network weight distribution corresponds to a renormalization flow in the space of networks.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A statistical ensemble of neural networks can be described in terms of a quantum field theory (NN-QFT correspondence). The infinite-width limit is mapped to a free field theory, while finite N corrections are mapped to interactions. After reviewing the correspondence, we will describe how to implement renormalization in this context and discuss preliminary numerical results for translation-invariant kernels. A major outcome is that changing the standard deviation of the neural network weight distribution corresponds to a renormalization flow in the space of networks.

Related papers

• Learning Theory of Distribution Regression with Neural Networks [6.961253535504979]
  We establish an approximation theory and a learning theory of distribution regression via a fully connected neural network (FNN) In contrast to the classical regression methods, the input variables of distribution regression are probability measures.
  arXiv  Detail & Related papers  (2023-07-07T09:49:11Z)
• Interrelation of equivariant Gaussian processes and convolutional neural networks [77.34726150561087]
  Currently there exists rather promising new trend in machine leaning (ML) based on the relationship between neural networks (NN) and Gaussian processes (GP) In this work we establish a relationship between the many-channel limit for CNNs equivariant with respect to two-dimensional Euclidean group with vector-valued neuron activations and the corresponding independently introduced equivariant Gaussian processes (GP)
  arXiv  Detail & Related papers  (2022-09-17T17:02:35Z)
• Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study [55.12108376616355]
  The study on NTK has been devoted to typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp) In this work, we derive the finite-width-K formulation for a special class of NNs-Hp, i.e., neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK.
  arXiv  Detail & Related papers  (2022-09-16T06:36:06Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Approximation bounds for norm constrained neural networks with applications to regression and GANs [9.645327615996914]
  We prove upper and lower bounds on the approximation error of ReLU neural networks with norm constraint on the weights. We apply these approximation bounds to analyze the convergences of regression using norm constrained neural networks and distribution estimation by GANs.
  arXiv  Detail & Related papers  (2022-01-24T02:19:05Z)
• The edge of chaos: quantum field theory and deep neural networks [0.0]
  We explicitly construct the quantum field theory corresponding to a general class of deep neural networks. We compute the loop corrections to the correlation function in a perturbative expansion in the ratio of depth $T$ to width $N$. Our analysis provides a first-principles approach to the rapidly emerging NN-QFT correspondence, and opens several interesting avenues to the study of criticality in deep neural networks.
  arXiv  Detail & Related papers  (2021-09-27T18:00:00Z)
• Nonperturbative renormalization for the neural network-QFT correspondence [0.0]
  We study the concepts of locality and power-counting in this context. We provide an analysis in terms of the nonperturbative renormalization group using the Wetterich-Morris equation. Our aim is to provide a useful formalism to investigate neural networks behavior beyond the large-width limit.
  arXiv  Detail & Related papers  (2021-08-03T10:36:04Z)
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks [56.084798078072396]
  We take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time. We show that convergence to a global minimum is guaranteed for networks with quadratic widths in the sample size and linear in their depth at a time logarithmic in both. Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size.
  arXiv  Detail & Related papers  (2021-01-12T00:40:45Z)
• Neural Networks and Quantum Field Theory [0.0]
  We propose a theoretical understanding of neural networks in terms of Wilsonian effective field theory. The correspondence relies on the fact that many neural networks are drawn from Gaussian processes.
  arXiv  Detail & Related papers  (2020-08-19T18:00:06Z)
• Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics [50.83356836818667]
  We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
  arXiv  Detail & Related papers  (2020-07-11T18:19:50Z)
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers [82.61117235807606]
  The success of deep neural networks in part due to the use of normalization layers. Our analysis shows how the introduction of normalization changes the landscape and can enable faster activation.
  arXiv  Detail & Related papers  (2020-06-11T23:55:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.