A connection between probability, physics and neural networks

• URL: http://arxiv.org/abs/2209.12737v1
• Date: Mon, 26 Sep 2022 14:40:09 GMT
• Title: A connection between probability, physics and neural networks
• Authors: Sascha Ranftl
• Abstract summary: We illustrate an approach that can be exploited for constructing neural networks which a priori obeys physical laws. We start with a simple single-layer neural network (NN) but refrain from choosing the activation functions yet. The activation functions constructed in this way guarantee the NN to a priori obey the physics, up to the approximation error of non-infinite network width.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We illustrate an approach that can be exploited for constructing neural networks which a priori obey physical laws. We start with a simple single-layer neural network (NN) but refrain from choosing the activation functions yet. Under certain conditions and in the infinite-width limit, we may apply the central limit theorem, upon which the NN output becomes Gaussian. We may then investigate and manipulate the limit network by falling back on Gaussian process (GP) theory. It is observed that linear operators acting upon a GP again yield a GP. This also holds true for differential operators defining differential equations and describing physical laws. If we demand the GP, or equivalently the limit network, to obey the physical law, then this yields an equation for the covariance function or kernel of the GP, whose solution equivalently constrains the model to obey the physical law. The central limit theorem then suggests that NNs can be constructed to obey a physical law by choosing the activation functions such that they match a particular kernel in the infinite-width limit. The activation functions constructed in this way guarantee the NN to a priori obey the physics, up to the approximation error of non-infinite network width. Simple examples of the homogeneous 1D-Helmholtz equation are discussed and compared to naive kernels and activations.

Related papers

• Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured? [4.878983382452911]
  We show that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) can only be approximated by a finite-dimensional vector space. We introduce the concepts of textit$epsilon$ outer measure and textitNumerical Span Dimension (NSdim) to quantify the approximation capacity limit of a family of networks.
  arXiv  Detail & Related papers  (2024-09-25T07:43:48Z)
• Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regime [52.00917519626559]
  This paper presents two models of neural-networks and their training applicable to neural networks of arbitrary width, depth and topology. We also present an exact novel representor theory for layer-wise neural network training with unregularized gradient descent in terms of a local-extrinsic neural kernel (LeNK) This representor theory gives insight into the role of higher-order statistics in neural network training and the effect of kernel evolution in neural-network kernel models.
  arXiv  Detail & Related papers  (2024-05-24T06:30:36Z)
• Multi-layer random features and the approximation power of neural networks [4.178980693837599]
  We prove that a reproducing kernel Hilbert space contains only functions that can be approximated by the architecture. We show that if eigenvalues of the integral operator of the NNGP decay slower than $k-n-frac23$ where $k$ is an order of an eigenvalue, our theorem guarantees a more succinct neural network approximation than Barron's theorem.
  arXiv  Detail & Related papers  (2024-04-26T14:57:56Z)
• Small-time controllability for the nonlinear Schr\"odinger equation on $\mathbb{R}^N$ via bilinear electromagnetic fields [55.2480439325792]
  We address the small-time controllability problem for a nonlinear Schr"odinger equation (NLS) on $mathbbRN$ in the presence of magnetic and electric external fields. In detail, we study when it is possible to control the dynamics of (NLS) as fast as desired via sufficiently large control signals.
  arXiv  Detail & Related papers  (2023-07-28T21:30:44Z)
• Speed Limits for Deep Learning [67.69149326107103]
  Recent advancement in thermodynamics allows bounding the speed at which one can go from the initial weight distribution to the final distribution of the fully trained network. We provide analytical expressions for these speed limits for linear and linearizable neural networks. Remarkably, given some plausible scaling assumptions on the NTK spectra and spectral decomposition of the labels -- learning is optimal in a scaling sense.
  arXiv  Detail & Related papers  (2023-07-27T06:59:46Z)
• Neural Network Field Theories: Non-Gaussianity, Actions, and Locality [0.0]
  Both the path integral measure in field theory and ensembles of neural networks describe distributions over functions. An expansion in $1/N$ corresponds to interactions in the field theory, but others, such as in a small breaking of the statistical independence of network parameters, can also lead to interacting theories.
  arXiv  Detail & Related papers  (2023-07-06T18:00:01Z)
• A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks [49.870593940818715]
  We study the infinite-width limit of a type of three-layer NN model whose first layer is random and fixed. Our theory accommodates different scaling choices of the model, resulting in two regimes of the MF limit that demonstrate distinctive behaviors.
  arXiv  Detail & Related papers  (2022-10-28T17:26:27Z)
• Sample-Then-Optimize Batch Neural Thompson Sampling [50.800944138278474]
  We introduce two algorithms for black-box optimization based on the Thompson sampling (TS) policy. To choose an input query, we only need to train an NN and then choose the query by maximizing the trained NN. Our algorithms sidestep the need to invert the large parameter matrix yet still preserve the validity of the TS policy.
  arXiv  Detail & Related papers  (2022-10-13T09:01:58Z)
• On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks [91.3755431537592]
  We study how random pruning of the weights affects a neural network's neural kernel (NTK) In particular, this work establishes an equivalence of the NTKs between a fully-connected neural network and its randomly pruned version.
  arXiv  Detail & Related papers  (2022-03-27T15:22:19Z)
• Nonperturbative renormalization for the neural network-QFT correspondence [0.0]
  We study the concepts of locality and power-counting in this context. We provide an analysis in terms of the nonperturbative renormalization group using the Wetterich-Morris equation. Our aim is to provide a useful formalism to investigate neural networks behavior beyond the large-width limit.
  arXiv  Detail & Related papers  (2021-08-03T10:36:04Z)
• Neural Networks and Quantum Field Theory [0.0]
  We propose a theoretical understanding of neural networks in terms of Wilsonian effective field theory. The correspondence relies on the fact that many neural networks are drawn from Gaussian processes.
  arXiv  Detail & Related papers  (2020-08-19T18:00:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.