Gradient-enhanced deep neural network approximations

• URL: http://arxiv.org/abs/2211.04226v1
• Date: Tue, 8 Nov 2022 13:16:02 GMT
• Title: Gradient-enhanced deep neural network approximations
• Authors: Xiaodong Feng, Li Zeng
• Abstract summary: gradient-enhanced deep neural networks (DNNs) approach for function approximations and uncertainty quantification. We present several numerical experiments to show that the proposed approach can outperform the traditional DNNs approach in many cases of interests.
• Score: 1.9721888064019697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose in this work the gradient-enhanced deep neural networks (DNNs) approach for function approximations and uncertainty quantification. More precisely, the proposed approach adopts both the function evaluations and the associated gradient information to yield enhanced approximation accuracy. In particular, the gradient information is included as a regularization term in the gradient-enhanced DNNs approach, for which we present similar posterior estimates (by the two-layer neural networks) as those in the path-norm regularized DNNs approximations. We also discuss the application of this approach to gradient-enhanced uncertainty quantification, and present several numerical experiments to show that the proposed approach can outperform the traditional DNNs approach in many cases of interests.

Related papers

• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks [59.142826407441106]
  We study the generalization behavior of shallow neural networks (SNNs) by leveraging the concept of algorithmic stability. We consider gradient descent (GD) and gradient descent (SGD) to train SNNs, for both of which we develop consistent excess bounds.
  arXiv  Detail & Related papers  (2022-09-19T18:48:00Z)
• Graph Neural Network Based Node Deployment for Throughput Enhancement [20.56966053013759]
  We propose a novel graph neural network (GNN) method for the network node deployment problem. We show that an expressive GNN has the capacity to approximate both the function value and the traffic permutation, as a theoretic support for the proposed method.
  arXiv  Detail & Related papers  (2022-08-19T08:06:28Z)
• A Differentiable Point Process with Its Application to Spiking Neural Networks [13.160616423673373]
  Jimenez Rezende & Gerstner (2014) proposed a variational inference algorithm to train SNNs with hidden neurons. This paper presents an alternative gradient estimator for SNNs based on the path-wise gradient estimator.
  arXiv  Detail & Related papers  (2021-06-02T02:40:17Z)
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime [50.510421854168065]
  We show that the averaged gradient descent can achieve the minimax optimal convergence rate. We show that the target function specified by the NTK of a ReLU network can be learned at the optimal convergence rate.
  arXiv  Detail & Related papers  (2020-06-22T14:31:37Z)
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks [78.76880041670904]
  In neural networks with binary activations and or binary weights the training by gradient descent is complicated. We propose a new method for this estimation problem combining sampling and analytic approximation steps. We experimentally show higher accuracy in gradient estimation and demonstrate a more stable and better performing training in deep convolutional models.
  arXiv  Detail & Related papers  (2020-06-04T21:51:21Z)
• Constructing Accurate and Efficient Deep Spiking Neural Networks with Double-threshold and Augmented Schemes [35.395895930338455]
  Spiking neural networks (SNNs) are considered as a potential candidate to overcome current challenges such as the high-power consumption encountered by artificial neural networks (ANNs)
  arXiv  Detail & Related papers  (2020-05-05T06:44:05Z)
• Tune smarter not harder: A principled approach to tuning learning rates for shallow nets [13.203765985718201]
  principled approach to choosing the learning rate is proposed for shallow feedforward neural networks. It is shown through simulations that the proposed search method significantly outperforms the existing tuning methods.
  arXiv  Detail & Related papers  (2020-03-22T09:38:35Z)
• Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs [71.26657499537366]
  We propose a simple literature-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method to train neural ODEs on classification, density estimation, and inference approximation tasks.
  arXiv  Detail & Related papers  (2020-03-11T13:15:57Z)
• Disentangling Adaptive Gradient Methods from Learning Rates [65.0397050979662]
  We take a deeper look at how adaptive gradient methods interact with the learning rate schedule. We introduce a "grafting" experiment which decouples an update's magnitude from its direction. We present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods.
  arXiv  Detail & Related papers  (2020-02-26T21:42:49Z)
• Semi-Implicit Back Propagation [1.5533842336139065]
  We propose a semi-implicit back propagation method for neural network training. The difference on the neurons are propagated in a backward fashion and the parameters are updated with proximal mapping. Experiments on both MNIST and CIFAR-10 demonstrate that the proposed algorithm leads to better performance in terms of both loss decreasing and training/validation accuracy.
  arXiv  Detail & Related papers  (2020-02-10T03:26:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.