Quantitative Flow Approximation Properties of Narrow Neural ODEs

• URL: http://arxiv.org/abs/2503.04068v1
• Date: Thu, 06 Mar 2025 03:54:42 GMT
• Title: Quantitative Flow Approximation Properties of Narrow Neural ODEs
• Authors: Karthik Elamvazhuthi,
• Abstract summary: We revisit the problem of flow approximation properties of neural ordinary differential equations (NODEs)<n>We derive the relation of narrow NODEs in approximating flows of shallow but wide NODEs.<n>We provide an estimate on the number of switches needed for the time dependent weights of the narrow NODE to mimic the behavior of a NODE with a single layer wide neural network as the velocity field.
• Score: 0.5439020425819
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this note, we revisit the problem of flow approximation properties of neural ordinary differential equations (NODEs). The approximation properties have been considered as a flow controllability problem in recent literature. The neural ODE is considered {\it narrow} when the parameters have dimension equal to the input of the neural network, and hence have limited width. We derive the relation of narrow NODEs in approximating flows of shallow but wide NODEs. Due to existing results on approximation properties of shallow neural networks, this facilitates understanding which kind of flows of dynamical systems can be approximated using narrow neural ODEs. While approximation properties of narrow NODEs have been established in literature, the proofs often involve extensive constructions or require invoking deep controllability theorems from control theory. In this paper, we provide a simpler proof technique that involves only ideas from ODEs and Gr{\"o}nwall's lemma. Moreover, we provide an estimate on the number of switches needed for the time dependent weights of the narrow NODE to mimic the behavior of a NODE with a single layer wide neural network as the velocity field.

Related papers

• Approximation properties of neural ODEs [3.6779367026247627]
  We study the approximation properties of shallow neural networks whose activation function is defined as the flow of a neural ordinary differential equation (neural ODE) We prove the universal approximation property (UAP) of such shallow neural networks in the space of continuous functions.
  arXiv  Detail & Related papers  (2025-03-19T21:11:28Z)
• Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured? [4.878983382452911]
  We show that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) can only be approximated by a finite-dimensional vector space. We introduce the concepts of textit$epsilon$ outer measure and textitNumerical Span Dimension (NSdim) to quantify the approximation capacity limit of a family of networks.
  arXiv  Detail & Related papers  (2024-09-25T07:43:48Z)
• From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks [68.8204255655161]
  We propose a new type of continuous-time control system, called AutoencODE, based on a controlled field that drives dynamics. We show that many architectures can be recovered in regions where the loss function is locally convex.
  arXiv  Detail & Related papers  (2023-07-05T13:26:17Z)
• Variational formulations of ODE-Net as a mean-field optimal control problem and existence results [0.0]
  This paper presents a mathematical analysis of ODE-Net, a continuum model of deep neural networks (DNNs) In recent years, Machine Learning researchers have introduced ideas of replacing the deep structure of DNNs with ODEs as a continuum limit.
  arXiv  Detail & Related papers  (2023-03-09T00:47:30Z)
• Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets [57.06026574261203]
  We provide theoretical evidence for learning low-dimensional state spaces, which can also model long-term memory. Experiments corroborate our theory, demonstrating extrapolation via learning low-dimensional state spaces with both linear and non-linear RNNs.
  arXiv  Detail & Related papers  (2022-10-25T14:45:15Z)
• Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study [55.12108376616355]
  The study on NTK has been devoted to typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp) In this work, we derive the finite-width-K formulation for a special class of NNs-Hp, i.e., neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK.
  arXiv  Detail & Related papers  (2022-09-16T06:36:06Z)
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis [94.64007376939735]
  We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training. We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
  arXiv  Detail & Related papers  (2022-05-11T17:43:54Z)
• On the Turnpike to Design of Deep Neural Nets: Explicit Depth Bounds [0.0]
  This paper attempts a quantifiable answer to the question of how many layers should be considered in a Deep Neural Networks (DNN) The underlying assumption is that the number of neurons per layer -- i.e., the width of the DNN -- is kept constant. We prove explicit bounds on the required depths of DNNs based on reachability of assumptions and a dissipativity-inducing choice of the regularization terms in the training problem.
  arXiv  Detail & Related papers  (2021-01-08T13:23:37Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• An Ode to an ODE [78.97367880223254]
  We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the group O(d) This nested system of two flows provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem.
  arXiv  Detail & Related papers  (2020-06-19T22:05:19Z)
• Go with the Flow: Adaptive Control for Neural ODEs [10.265713480189484]
  We describe a new module called neurally controlled ODE (N-CODE) designed to improve the expressivity of NODEs. N-CODE modules are dynamic variables governed by a trainable map from initial or current activation state. A single module is sufficient for learning a distribution on non-autonomous flows that adaptively drive neural representations.
  arXiv  Detail & Related papers  (2020-06-16T22:21:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.