Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data

• URL: http://arxiv.org/abs/2012.06081v2
• Date: Fri, 5 Mar 2021 00:48:51 GMT
• Title: Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data
• Authors: Ben Adcock and Simone Brugiapaglia and Nick Dexter and Sebastian Moraga
• Abstract summary: We focus on approximating functions that are Hilbert-valued, i.e. take values in a separable, but typically infinite-dimensional, Hilbert space. We present a novel result on DNN training for holomorphic functions with so-called hidden anisotropy. We show that there exists a procedure for learning Hilbert-valued functions via DNNs that performs as well as, but no better than current best-in-class schemes.
• Score: 6.098254376499899
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate approximation of scalar-valued functions from sample points is a key task in computational science. Recently, machine learning with Deep Neural Networks (DNNs) has emerged as a promising tool for scientific computing, with impressive results achieved on problems where the dimension of the data or problem domain is large. This work broadens this perspective, focusing on approximating functions that are Hilbert-valued, i.e. take values in a separable, but typically infinite-dimensional, Hilbert space. This arises in science and engineering problems, in particular those involving solution of parametric Partial Differential Equations (PDEs). Such problems are challenging: 1) pointwise samples are expensive to acquire, 2) the function domain is high dimensional, and 3) the range lies in a Hilbert space. Our contributions are twofold. First, we present a novel result on DNN training for holomorphic functions with so-called hidden anisotropy. This result introduces a DNN training procedure and full theoretical analysis with explicit guarantees on error and sample complexity. The error bound is explicit in three key errors occurring in the approximation procedure: the best approximation, measurement, and physical discretization errors. Our result shows that there exists a procedure (albeit non-standard) for learning Hilbert-valued functions via DNNs that performs as well as, but no better than current best-in-class schemes. It gives a benchmark lower bound for how well DNNs can perform on such problems. Second, we examine whether better performance can be achieved in practice through different types of architectures and training. We provide preliminary numerical results illustrating practical performance of DNNs on parametric PDEs. We consider different parameters, modifying the DNN architecture to achieve better and competitive results, comparing these to current best-in-class schemes.

Related papers

• Improving PINNs By Algebraic Inclusion of Boundary and Initial Conditions [0.1874930567916036]
  "AI for Science" aims to solve fundamental scientific problems using AI techniques. In this work we explore the possibility of changing the model being trained from being just a neural network to being a non-linear transformation of it. This reduces the number of terms in the loss function than the standard PINN losses.
  arXiv  Detail & Related papers  (2024-07-30T11:19:48Z)
• Optimal deep learning of holomorphic operators between Banach spaces [0.6554326244334866]
  We tackle the problem of learning operators between Banach spaces, in contrast to the vast majority of past works considering only Hilbert spaces. We combine arbitrary approximate encoders and decoders with standard feedforward Deep Neural Network (DNN) architectures. We show that DL is optimal for this problem: no recovery procedure can surpass these generalization bounds up to log terms.
  arXiv  Detail & Related papers  (2024-06-20T01:49:42Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• 0/1 Deep Neural Networks via Block Coordinate Descent [40.11141921215105]
  The step function is one of the simplest and most natural activation functions for deep neural networks (DNNs) As it counts 1 for positive variables and 0 for others, its intrinsic characteristics (e.g., discontinuity and no viable information of subgradients) impede its development for decades.
  arXiv  Detail & Related papers  (2022-06-19T11:12:30Z)
• Characterizing possible failure modes in physics-informed neural networks [55.83255669840384]
  Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena even for simple PDEs. We show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize.
  arXiv  Detail & Related papers  (2021-09-02T16:06:45Z)
• dNNsolve: an efficient NN-based PDE solver [62.997667081978825]
  We introduce dNNsolve, that makes use of dual Neural Networks to solve ODEs/PDEs. We show that dNNsolve is capable of solving a broad range of ODEs/PDEs in 1, 2 and 3 spacetime dimensions.
  arXiv  Detail & Related papers  (2021-03-15T19:14:41Z)
• Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection [2.7561479348365734]
  Deep neural networks (DNNs) have achieved state-of-theart performance across a variety of traditional machine learning tasks. In this paper, we consider training of DNNs, which arises in many state-of-the-art applications.
  arXiv  Detail & Related papers  (2020-07-26T16:29:39Z)
• Fast Learning of Graph Neural Networks with Guaranteed Generalizability: One-hidden-layer Case [93.37576644429578]
  Graph neural networks (GNNs) have made great progress recently on learning from graph-structured data in practice. We provide a theoretically-grounded generalizability analysis of GNNs with one hidden layer for both regression and binary classification problems.
  arXiv  Detail & Related papers  (2020-06-25T00:45:52Z)
• Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks [60.22494363676747]
  It is known that the current graph neural networks (GNNs) are difficult to make themselves deep due to the problem known as over-smoothing. Multi-scale GNNs are a promising approach for mitigating the over-smoothing problem. We derive the optimization and generalization guarantees of transductive learning algorithms that include multi-scale GNNs.
  arXiv  Detail & Related papers  (2020-06-15T17:06:17Z)
• Self-Directed Online Machine Learning for Topology Optimization [58.920693413667216]
  Self-directed Online Learning Optimization integrates Deep Neural Network (DNN) with Finite Element Method (FEM) calculations. Our algorithm was tested by four types of problems including compliance minimization, fluid-structure optimization, heat transfer enhancement and truss optimization. It reduced the computational time by 2 5 orders of magnitude compared with directly using methods, and outperformed all state-of-the-art algorithms tested in our experiments.
  arXiv  Detail & Related papers  (2020-02-04T20:00:28Z)
• The gap between theory and practice in function approximation with deep neural networks [2.969705152497174]
  Deep learning (DL) is transforming industry as decision-making processes are being automated by deep neural networks (DNNs) trained on real-world data. We introduce a computational framework for examining DNNs in practice, and use it to study empirical performance with regard to these issues.
  arXiv  Detail & Related papers  (2020-01-16T20:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.