Numerical and statistical analysis of NeuralODE with Runge-Kutta time integration

• URL: http://arxiv.org/abs/2503.10729v1
• Date: Thu, 13 Mar 2025 11:58:18 GMT
• Title: Numerical and statistical analysis of NeuralODE with Runge-Kutta time integration
• Authors: Emily C. Ehrhardt, Hanno Gottschalk, Tobias J. Riedlinger,
• Abstract summary: We give a detailed account on the consistency of Maximum Likelihood based empirical risk minimization for a generic class of target measures.<n>We also give a numerical analysis of the NeuralODE algorithm based on the 2nd order Runge-Kutta (RK) time integration.
• Score: 1.3654846342364308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: NeuralODE is one example for generative machine learning based on the push forward of a simple source measure with a bijective mapping, which in the case of NeuralODE is given by the flow of a ordinary differential equation. Using Liouville's formula, the log-density of the push forward measure is easy to compute and thus NeuralODE can be trained based on the maximum Likelihood method such that the Kulback-Leibler divergence between the push forward through the flow map and the target measure generating the data becomes small. In this work, we give a detailed account on the consistency of Maximum Likelihood based empirical risk minimization for a generic class of target measures. In contrast to prior work, we do not only consider the statistical learning theory, but also give a detailed numerical analysis of the NeuralODE algorithm based on the 2nd order Runge-Kutta (RK) time integration. Using the universal approximation theory for deep ReQU networks, the stability and convergence rated for the RK scheme as well as metric entropy and concentration inequalities, we are able to prove that NeuralODE is a probably approximately correct (PAC) learning algorithm.

Related papers

• On the Wasserstein Convergence and Straightness of Rectified Flow [54.580605276017096]
  Rectified Flow (RF) is a generative model that aims to learn straight flow trajectories from noise to data. We provide a theoretical analysis of the Wasserstein distance between the sampling distribution of RF and the target distribution. We present general conditions guaranteeing uniqueness and straightness of 1-RF, which is in line with previous empirical findings.
  arXiv  Detail & Related papers  (2024-10-19T02:36:11Z)
• Convergence Analysis for General Probability Flow ODEs of Diffusion Models in Wasserstein Distances [9.47767039367222]
  We provide the first non-asymptotic convergence analysis for a general class of probability flow ODE samplers in 2-Wasserstein distance. Our proof technique relies on spelling out explicitly the contraction rate for the continuous-time ODE and analyzing the discretization and score-matching errors using synchronous coupling.
  arXiv  Detail & Related papers  (2024-01-31T16:07:44Z)
• Distribution learning via neural differential equations: a nonparametric statistical perspective [1.4436965372953483]
  This work establishes the first general statistical convergence analysis for distribution learning via ODE models trained through likelihood transformations. We show that the latter can be quantified via the $C1$-metric entropy of the class $mathcal F$. We then apply this general framework to the setting of $Ck$-smooth target densities, and establish nearly minimax-optimal convergence rates for two relevant velocity field classes $mathcal F$: $Ck$ functions and neural networks.
  arXiv  Detail & Related papers  (2023-09-03T00:21:37Z)
• Reflected Diffusion Models [93.26107023470979]
  We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data. Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
  arXiv  Detail & Related papers  (2023-04-10T17:54:38Z)
• D4FT: A Deep Learning Approach to Kohn-Sham Density Functional Theory [79.50644650795012]
  We propose a deep learning approach to solve Kohn-Sham Density Functional Theory (KS-DFT) We prove that such an approach has the same expressivity as the SCF method, yet reduces the computational complexity. In addition, we show that our approach enables us to explore more complex neural-based wave functions.
  arXiv  Detail & Related papers  (2023-03-01T10:38:10Z)
• Robust and Provably Monotonic Networks [0.0]
  We present a new method to constrain the Lipschitz constant of dense deep learning models. We show how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor decays in the LHCb realtime data-processing system.
  arXiv  Detail & Related papers  (2021-11-30T19:01:32Z)
• Distributional Gradient Matching for Learning Uncertain Neural Dynamics Models [38.17499046781131]
  We propose a novel approach towards estimating uncertain neural ODEs, avoiding the numerical integration bottleneck. Our algorithm - distributional gradient matching (DGM) - jointly trains a smoother and a dynamics model and matches their gradients via minimizing a Wasserstein loss. Our experiments show that, compared to traditional approximate inference methods based on numerical integration, our approach is faster to train, faster at predicting previously unseen trajectories, and in the context of neural ODEs, significantly more accurate.
  arXiv  Detail & Related papers  (2021-06-22T08:40:51Z)
• Influence Estimation and Maximization via Neural Mean-Field Dynamics [60.91291234832546]
  We propose a novel learning framework using neural mean-field (NMF) dynamics for inference and estimation problems. Our framework can simultaneously learn the structure of the diffusion network and the evolution of node infection probabilities.
  arXiv  Detail & Related papers  (2021-06-03T00:02:05Z)
• Bayesian Neural Ordinary Differential Equations [0.9422623204346027]
  We demonstrate the successful integration of Neural ODEs with Bayesian inference frameworks. We achieve a posterior sample accuracy of 98.5% on the test ensemble of 10,000 images. This gives a scientific machine learning tool for probabilistic estimation of uncertainties.
  arXiv  Detail & Related papers  (2020-12-14T04:05:26Z)
• ResNet After All? Neural ODEs and Their Numerical Solution [28.954378025052925]
  We show that trained Neural Ordinary Differential Equation models actually depend on the specific numerical method used during training. We propose a method that monitors the behavior of the ODE solver during training to adapt its step size.
  arXiv  Detail & Related papers  (2020-07-30T11:24:05Z)
• Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs [71.26657499537366]
  We propose a simple literature-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method to train neural ODEs on classification, density estimation, and inference approximation tasks.
  arXiv  Detail & Related papers  (2020-03-11T13:15:57Z)
• SLEIPNIR: Deterministic and Provably Accurate Feature Expansion for Gaussian Process Regression with Derivatives [86.01677297601624]
  We propose a novel approach for scaling GP regression with derivatives based on quadrature Fourier features. We prove deterministic, non-asymptotic and exponentially fast decaying error bounds which apply for both the approximated kernel as well as the approximated posterior.
  arXiv  Detail & Related papers  (2020-03-05T14:33:20Z)
• Stochasticity in Neural ODEs: An Empirical Study [68.8204255655161]
  Regularization of neural networks (e.g. dropout) is a widespread technique in deep learning that allows for better generalization. We show that data augmentation during the training improves the performance of both deterministic and versions of the same model. However, the improvements obtained by the data augmentation completely eliminate the empirical regularization gains, making the performance of neural ODE and neural SDE negligible.
  arXiv  Detail & Related papers  (2020-02-22T22:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.