Approximation Properties of Deep ReLU CNNs

• URL: http://arxiv.org/abs/2109.00190v1
• Date: Wed, 1 Sep 2021 05:16:11 GMT
• Title: Approximation Properties of Deep ReLU CNNs
• Authors: Juncai He, Lin Li, Jinchao Xu
• Abstract summary: This paper is devoted to establishing $L2$ approximation properties for deep ReLU convolutional neural networks (CNNs) on two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels with large spatial size and multi-channel.
• Score: 8.74591882131599
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper is devoted to establishing $L^2$ approximation properties for deep ReLU convolutional neural networks (CNNs) on two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels with large spatial size and multi-channel. Given that decomposition and the property of the ReLU activation function, a universal approximation theorem of deep ReLU CNNs with classic structure is obtained by showing its connection with ReLU deep neural networks (DNNs) with one hidden layer. Furthermore, approximation properties are also obtained for neural networks with ResNet, pre-act ResNet, and MgNet architecture based on connections between these networks.

Related papers

• Residual resampling-based physics-informed neural network for neutron diffusion equations [7.105073499157097]
  The neutron diffusion equation plays a pivotal role in the analysis of nuclear reactors. Traditional PINN approaches often utilize fully connected network (FCN) architecture. R2-PINN effectively overcomes the limitations inherent in current methods, providing more accurate and robust solutions for neutron diffusion equations.
  arXiv  Detail & Related papers  (2024-06-23T13:49:31Z)
• On the growth of the parameters of approximating ReLU neural networks [0.542249320079018]
  This work focuses on the analysis of fully connected feed forward ReLU neural networks as they approximate a given, smooth function. In contrast to conventionally studied universal approximation properties under increasing architectures, we are concerned with the growth of the parameters of approximating networks.
  arXiv  Detail & Related papers  (2024-06-21T07:45:28Z)
• Data Topology-Dependent Upper Bounds of Neural Network Widths [52.58441144171022]
  We first show that a three-layer neural network can be designed to approximate an indicator function over a compact set. This is then extended to a simplicial complex, deriving width upper bounds based on its topological structure. We prove the universal approximation property of three-layer ReLU networks using our topological approach.
  arXiv  Detail & Related papers  (2023-05-25T14:17:15Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study [55.12108376616355]
  The study on NTK has been devoted to typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp) In this work, we derive the finite-width-K formulation for a special class of NNs-Hp, i.e., neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK.
  arXiv  Detail & Related papers  (2022-09-16T06:36:06Z)
• Towards Understanding Theoretical Advantages of Complex-Reaction Networks [77.34726150561087]
  We show that a class of functions can be approximated by a complex-reaction network using the number of parameters. For empirical risk minimization, our theoretical result shows that the critical point set of complex-reaction networks is a proper subset of that of real-valued networks.
  arXiv  Detail & Related papers  (2021-08-15T10:13:49Z)
• ReLU Deep Neural Networks from the Hierarchical Basis Perspective [8.74591882131599]
  We study ReLU deep neural networks (DNNs) by investigating their connections with the hierarchical basis method in finite element methods. We show that the approximation schemes of ReLU DNNs for $x2$ and $xy$ are composition versions of the hierarchical basis approximation for these two functions.
  arXiv  Detail & Related papers  (2021-05-10T07:25:33Z)
• Kernel-Based Smoothness Analysis of Residual Networks [85.20737467304994]
  Residual networks (ResNets) stand out among these powerful modern architectures. In this paper, we show another distinction between the two models, namely, a tendency of ResNets to promote smoothers than gradients.
  arXiv  Detail & Related papers  (2020-09-21T16:32:04Z)
• Expressivity of Deep Neural Networks [2.7909470193274593]
  In this review paper, we give a comprehensive overview of the large variety of approximation results for neural networks. While the mainbody of existing results is for general feedforward architectures, we also depict approximation results for convolutional, residual and recurrent neural networks.
  arXiv  Detail & Related papers  (2020-07-09T13:08:01Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• Approximation in shift-invariant spaces with deep ReLU neural networks [7.7084107194202875]
  We study the expressive power of deep ReLU neural networks for approximating functions in dilated shift-invariant spaces. Approximation error bounds are estimated with respect to the width and depth of neural networks.
  arXiv  Detail & Related papers  (2020-05-25T07:23:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.