Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equations

• URL: http://arxiv.org/abs/2603.18331v1
• Date: Wed, 18 Mar 2026 22:41:02 GMT
• Title: Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equations
• Abstract summary: We present differential equations as a theoretical foundation for understanding, analyzing, and improving Deep Neural Networks (DNNs)<n>We adopt a two-fold perspective spanning the model level, which interprets the whole DNN as a differential equation, and the layer level, which models individual DNN components as differential equations.<n>From these two perspectives, we review how this framework connects model design, theoretical analysis, and performance improvement.
• Score: 22.671087929643466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks (DNNs) have achieved remarkable empirical success, yet the absence of a principled theoretical foundation continues to hinder their systematic development. In this survey, we present differential equations as a theoretical foundation for understanding, analyzing, and improving DNNs. We organize the discussion around three guiding questions: i) how differential equations offer a principled understanding of DNN architectures, ii) how tools from differential equations can be used to improve DNN performance in a principled way, and iii) what real-world applications benefit from grounding DNNs in differential equations. We adopt a two-fold perspective spanning the model level, which interprets the whole DNN as a differential equation, and the layer level, which models individual DNN components as differential equations. From these two perspectives, we review how this framework connects model design, theoretical analysis, and performance improvement. We further discuss real-world applications, as well as key challenges and opportunities for future research.

Related papers

• Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs [94.85514873380572]
  This feature article provides an overview of the theoretical foundations for coVariance neural networks (VNNs)<n>Existing theoretical contributions on graph neural networks (GNNs) consider abstract graph representations.<n>This tutorial brings into focus various novel theoretical insights via mathematical analyses of VNNs that have broad signal processing implications.
  arXiv  Detail & Related papers  (2026-09-09T17:28:46Z)
• Expanding the Chaos: Neural Operator for Stochastic (Partial) Differential Equations [65.80144621950981]
  We build on Wiener chaos expansions (WCE) to design neural operator (NO) architectures for SPDEs and SDEs.<n>We show that WCE-based neural operators provide a practical and scalable way to learn SDE/SPDE solution operators.
  arXiv  Detail & Related papers  (2026-01-03T00:59:25Z)
• Deep Neural Networks Inspired by Differential Equations [32.63347759387102]
  Deep learning has become a pivotal technology in fields such as computer vision, scientific computing, and dynamical systems.<n> neural Networks persistently face challenges related to theoretical understanding, interpretability, and generalization.<n>We provide an extensive review of deep neural network and dynamic modeling methods inspired by differential equations.
  arXiv  Detail & Related papers  (2025-10-09T04:08:23Z)
• Towards the Dynamics of a DNN Learning Symbolic Interactions [20.493304123269446]
  A series of theorems have been proven in recent years to show that for a given input sample, a small set of interactions between input variables can be considered as primitive inference patterns. This study proves the two-phase dynamics of interactions, providing a theoretical mechanism for how the generalization power of a DNN changes during the training process.
  arXiv  Detail & Related papers  (2024-07-27T07:34:49Z)
• Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks [50.29356570858905]
  We introduce the Dynamical Systems Framework (DSF), which allows a principled investigation of all these architectures in a common representation.<n>We provide principled comparisons between softmax attention and other model classes, discussing the theoretical conditions under which softmax attention can be approximated.<n>This shows the DSF's potential to guide the systematic development of future more efficient and scalable foundation models.
  arXiv  Detail & Related papers  (2024-05-24T17:19:57Z)
• Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features [68.3512123520931]
  We investigate the dynamics of a deep neural network (DNN) learning interactions. In this paper, we discover the DNN learns interactions in two phases. The first phase mainly penalizes interactions of medium and high orders, and the second phase mainly learns interactions of gradually increasing orders.
  arXiv  Detail & Related papers  (2024-05-16T17:13:25Z)
• Pseudo-Hamiltonian neural networks for learning partial differential equations [0.0]
  Pseudo-Hamiltonian neural networks (PHNN) were recently introduced for learning dynamical systems that can be modelled by ordinary differential equations. In this paper, we extend the method to partial differential equations. The resulting model is comprised of up to three neural networks, modelling terms representing conservation, dissipation and external forces, and discrete convolution operators that can either be learned or be given as input.
  arXiv  Detail & Related papers  (2023-04-27T17:46:00Z)
• A Survey on Solving and Discovering Differential Equations Using Deep Neural Networks [1.0055663066199056]
  Ordinary and partial differential equations (DE) are used extensively in scientific and mathematical domains to model physical systems. Current literature has focused primarily on deep neural network (DNN) based methods for solving a specific DE or a family of DEs. This paper surveys and classifies previous works and provides an educational tutorial for senior practitioners, professionals, and graduate students in engineering and computer science.
  arXiv  Detail & Related papers  (2023-04-26T20:14:25Z)
• Quantum-Inspired Tensor Neural Networks for Partial Differential Equations [5.963563752404561]
  Deep learning methods are constrained by training time and memory. To tackle these shortcomings, we implement Neural Networks (TNN) We demonstrate that TNN provide significant parameter savings while attaining the same accuracy as compared to the classical Neural Network (DNN)
  arXiv  Detail & Related papers  (2022-08-03T17:41:11Z)
• Knowledge Enhanced Neural Networks for relational domains [83.9217787335878]
  We focus on a specific method, KENN, a Neural-Symbolic architecture that injects prior logical knowledge into a neural network. In this paper, we propose an extension of KENN for relational data.
  arXiv  Detail & Related papers  (2022-05-31T13:00:34Z)
• Universal approximation property of invertible neural networks [76.95927093274392]
  Invertible neural networks (INNs) are neural network architectures with invertibility by design. Thanks to their invertibility and the tractability of Jacobian, INNs have various machine learning applications such as probabilistic modeling, generative modeling, and representation learning.
  arXiv  Detail & Related papers  (2022-04-15T10:45:26Z)
• The gap between theory and practice in function approximation with deep neural networks [2.969705152497174]
  Deep learning (DL) is transforming industry as decision-making processes are being automated by deep neural networks (DNNs) trained on real-world data. We introduce a computational framework for examining DNNs in practice, and use it to study empirical performance with regard to these issues.
  arXiv  Detail & Related papers  (2020-01-16T20:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.