Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks

• URL: http://arxiv.org/abs/2404.03761v1
• Date: Thu, 4 Apr 2024 19:07:21 GMT
• Title: Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks
• Authors: Ben Adcock, Simone Brugiapaglia, Nick Dexter, Sebastian Moraga,
• Abstract summary: Learning approximations to smooth target functions of many variables from finite sets of pointwise samples is an important task in scientific computing. Significant advances have been made in the last decade towards efficient methods for doing this. Recent advances have been made in the relevant approximation theory and analysis of these techniques.
• Score: 0.9749638953163389
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning approximations to smooth target functions of many variables from finite sets of pointwise samples is an important task in scientific computing and its many applications in computational science and engineering. Despite well over half a century of research on high-dimensional approximation, this remains a challenging problem. Yet, significant advances have been made in the last decade towards efficient methods for doing this, commencing with so-called sparse polynomial approximation methods and continuing most recently with methods based on Deep Neural Networks (DNNs). In tandem, there have been substantial advances in the relevant approximation theory and analysis of these techniques. In this work, we survey this recent progress. We describe the contemporary motivations for this problem, which stem from parametric models and computational uncertainty quantification; the relevant function classes, namely, classes of infinite-dimensional, Banach-valued, holomorphic functions; fundamental limits of learnability from finite data for these classes; and finally, sparse polynomial and DNN methods for efficiently learning such functions from finite data. For the latter, there is currently a significant gap between the approximation theory of DNNs and the practical performance of deep learning. Aiming to narrow this gap, we develop the topic of practical existence theory, which asserts the existence of dimension-independent DNN architectures and training strategies that achieve provably near-optimal generalization errors in terms of the amount of training data.

Related papers

• An Overview on Machine Learning Methods for Partial Differential Equations: from Physics Informed Neural Networks to Deep Operator Learning [5.75055574132362]
  approximation of solutions of partial differential equations with numerical algorithms is a central topic in applied mathematics. One class of methods which has received a lot of attention in recent years are machine learning-based methods. This article aims to provide an introduction to some of these methods and the mathematical theory on which they are based.
  arXiv  Detail & Related papers  (2024-08-23T16:57:34Z)
• Scaling up Probabilistic PDE Simulators with Structured Volumetric Information [23.654711580674885]
  We propose a framework combining a discretization scheme based on the popular Finite Volume Method with complementary numerical linear algebra techniques. Experiments, including atemporal tsunami simulation, demonstrate substantially improved scaling behavior of this approach over previous collocation-based techniques.
  arXiv  Detail & Related papers  (2024-06-07T15:38:27Z)
• A practical existence theorem for reduced order models based on convolutional autoencoders [0.4604003661048266]
  Deep learning has gained increasing popularity in the fields of Partial Differential Equations (PDEs) and Reduced Order Modeling (ROM) CNN-based autoencoders have proven extremely effective, outperforming established techniques, such as the reduced basis method, when dealing with complex nonlinear problems. We provide a new practical existence theorem for CNN-based autoencoders when the parameter-to-solution map is holomorphic.
  arXiv  Detail & Related papers  (2024-02-01T09:01:58Z)
• Higher-order topological kernels via quantum computation [68.8204255655161]
  Topological data analysis (TDA) has emerged as a powerful tool for extracting meaningful insights from complex data. We propose a quantum approach to defining Betti kernels, which is based on constructing Betti curves with increasing order.
  arXiv  Detail & Related papers  (2023-07-14T14:48:52Z)
• Partial Differential Equations Meet Deep Neural Networks: A Survey [10.817323756266527]
  Problems in science and engineering can be represented by a set of partial differential equations (PDEs) through mathematical modeling. Mechanism-based computation following PDEs has long been an essential paradigm for studying topics such as computational fluid dynamics. Deep Neural Networks (DNNs) for PDEs have emerged as an effective means of solving PDEs.
  arXiv  Detail & Related papers  (2022-10-27T07:01:56Z)
• Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficient [65.08966446962845]
  offline reinforcement learning, which aims at optimizing decision-making strategies with historical data, has been extensively applied in real-life applications. We take a step by considering offline reinforcement learning with differentiable function class approximation (DFA) Most importantly, we show offline differentiable function approximation is provably efficient by analyzing the pessimistic fitted Q-learning algorithm.
  arXiv  Detail & Related papers  (2022-10-03T07:59:42Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces [208.67848059021915]
  We study the exploration-exploitation tradeoff at the core of reinforcement learning. In particular, we prove that the complexity of the function class $mathcalF$ characterizes the complexity of the function. Our regret bounds are independent of the number of episodes.
  arXiv  Detail & Related papers  (2020-11-09T18:32:22Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)
• Towards an Efficient and General Framework of Robust Training for Graph Neural Networks [96.93500886136532]
  Graph Neural Networks (GNNs) have made significant advances on several fundamental inference tasks. Despite GNNs' impressive performance, it has been observed that carefully crafted perturbations on graph structures lead them to make wrong predictions. We propose a general framework which leverages the greedy search algorithms and zeroth-order methods to obtain robust GNNs.
  arXiv  Detail & Related papers  (2020-02-25T15:17:58Z)
• The gap between theory and practice in function approximation with deep neural networks [2.969705152497174]
  Deep learning (DL) is transforming industry as decision-making processes are being automated by deep neural networks (DNNs) trained on real-world data. We introduce a computational framework for examining DNNs in practice, and use it to study empirical performance with regard to these issues.
  arXiv  Detail & Related papers  (2020-01-16T20:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.