Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems

• URL: http://arxiv.org/abs/2209.10909v3
• Date: Sun, 27 Oct 2024 06:56:09 GMT
• Title: Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems
• Authors: Yifei Duan, Li'ang Li, Guanghua Ji, Yongqiang Cai,
• Abstract summary: In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks.
• Score: 9.382423715831687
• License:
• Abstract: Deep learning has made significant applications in the field of data science and natural science. Some studies have linked deep neural networks to dynamic systems, but the network structure is restricted to the residual network. It is known that residual networks can be regarded as a numerical discretization of dynamic systems. In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems, where the width of the network is equal to the dimension of the input and output. Our proof is based on the properties of the leaky-ReLU function and the numerical technique of splitting method to solve differential equations. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks.

Related papers

• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Dynamical stability and chaos in artificial neural network trajectories along training [3.379574469735166]
  We study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network. We find hints of regular and chaotic behavior depending on the learning rate regime. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning.
  arXiv  Detail & Related papers  (2024-04-08T17:33:11Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• On the effectiveness of neural priors in modeling dynamical systems [28.69155113611877]
  We discuss the architectural regularization that neural networks offer when learning such systems. We show that simple coordinate networks with few layers can be used to solve multiple problems in modelling dynamical systems.
  arXiv  Detail & Related papers  (2023-03-10T06:21:24Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Learning Contact Dynamics using Physically Structured Neural Networks [81.73947303886753]
  We use connections between deep neural networks and differential equations to design a family of deep network architectures for representing contact dynamics between objects. We show that these networks can learn discontinuous contact events in a data-efficient manner from noisy observations. Our results indicate that an idealised form of touch feedback is a key component of making this learning problem tractable.
  arXiv  Detail & Related papers  (2021-02-22T17:33:51Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• Neural networks adapting to datasets: learning network size and topology [77.34726150561087]
  We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a gradient-based training. The resulting network has the structure of a graph tailored to the particular learning task and dataset.
  arXiv  Detail & Related papers  (2020-06-22T12:46:44Z)
• Input-to-State Representation in linear reservoirs dynamics [15.491286626948881]
  Reservoir computing is a popular approach to design recurrent neural networks. The working principle of these networks is not fully understood. A novel analysis of the dynamics of such networks is proposed.
  arXiv  Detail & Related papers  (2020-03-24T00:14:25Z)
• Mean-Field and Kinetic Descriptions of Neural Differential Equations [0.0]
  In this work we focus on a particular class of neural networks, i.e. the residual neural networks. We analyze steady states and sensitivity with respect to the parameters of the network, namely the weights and the bias. A modification of the microscopic dynamics, inspired by residual neural networks, leads to a Fokker-Planck formulation of the network.
  arXiv  Detail & Related papers  (2020-01-07T13:41:27Z)
• Emergence of Network Motifs in Deep Neural Networks [0.35911228556176483]
  We show that network science tools can be successfully applied to the study of artificial neural networks. In particular, we study the emergence of network motifs in multi-layer perceptrons.
  arXiv  Detail & Related papers  (2019-12-27T17:05:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.