ReLU Deep Neural Networks from the Hierarchical Basis Perspective

• URL: http://arxiv.org/abs/2105.04156v1
• Date: Mon, 10 May 2021 07:25:33 GMT
• Title: ReLU Deep Neural Networks from the Hierarchical Basis Perspective
• Authors: Juncai He, Lin Li, Jinchao Xu
• Abstract summary: We study ReLU deep neural networks (DNNs) by investigating their connections with the hierarchical basis method in finite element methods. We show that the approximation schemes of ReLU DNNs for $x2$ and $xy$ are composition versions of the hierarchical basis approximation for these two functions.
• Score: 8.74591882131599
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study ReLU deep neural networks (DNNs) by investigating their connections with the hierarchical basis method in finite element methods. First, we show that the approximation schemes of ReLU DNNs for $x^2$ and $xy$ are composition versions of the hierarchical basis approximation for these two functions. Based on this fact, we obtain a geometric interpretation and systematic proof for the approximation result of ReLU DNNs for polynomials, which plays an important role in a series of recent exponential approximation results of ReLU DNNs. Through our investigation of connections between ReLU DNNs and the hierarchical basis approximation for $x^2$ and $xy$, we show that ReLU DNNs with this special structure can be applied only to approximate quadratic functions. Furthermore, we obtain a concise representation to explicitly reproduce any linear finite element function on a two-dimensional uniform mesh by using ReLU DNNs with only two hidden layers.

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation [22.689531776611084]
  We present-augmented neural networks (PANNs) We present a novel machine learning architecture that combines deep neural networks (DNNs) with an approximant.
  arXiv  Detail & Related papers  (2024-06-04T14:06:15Z)
• On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem [0.0]
  We study the optimal expressive power of ReLU deep neural networks (DNNs) and its application in approximation via the Kolmogorov Superposition Theorem.
  arXiv  Detail & Related papers  (2023-08-10T11:42:09Z)
• Representation and decomposition of functions in DAG-DNNs and structural network pruning [3.04585143845864]
  We show that DAG-DNNs can be used to derive functions defined on various sub-architectures of a deep neural network (DNN) The lifting structure associated with lower-triangular matrices makes it possible to perform the structural pruning of a network in a systematic manner.
  arXiv  Detail & Related papers  (2023-06-16T09:18:36Z)
• Universal approximation property of invertible neural networks [76.95927093274392]
  Invertible neural networks (INNs) are neural network architectures with invertibility by design. Thanks to their invertibility and the tractability of Jacobian, INNs have various machine learning applications such as probabilistic modeling, generative modeling, and representation learning.
  arXiv  Detail & Related papers  (2022-04-15T10:45:26Z)
• Approximation Properties of Deep ReLU CNNs [8.74591882131599]
  This paper is devoted to establishing $L2$ approximation properties for deep ReLU convolutional neural networks (CNNs) on two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels with large spatial size and multi-channel.
  arXiv  Detail & Related papers  (2021-09-01T05:16:11Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time [70.15611146583068]
  We study training of Convolutional Neural Networks (CNNs) with ReLU activations. We introduce exact convex optimization with a complexity with respect to the number of data samples, the number of neurons, and data dimension.
  arXiv  Detail & Related papers  (2020-06-26T04:47:20Z)
• A Formal Hierarchy of RNN Architectures [88.38859874233944]
  hierarchy is based on two formal properties: space, which measures the RNN's memory, and rational recurrence, defined as whether the recurrent update can be described by a weighted finite-state machine. We show how these models' expressive capacity is expanded by stacking multiple layers or composing them with different pooling functions. We hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy.
  arXiv  Detail & Related papers  (2020-04-18T00:57:54Z)
• Interpretable Deep Recurrent Neural Networks via Unfolding Reweighted $\ell_1$-$\ell_1$ Minimization: Architecture Design and Generalization Analysis [19.706363403596196]
  This paper develops a novel deep recurrent neural network (coined reweighted-RNN) by the unfolding of a reweighted minimization algorithm. To the best of our knowledge, this is the first deep unfolding method that explores reweighted minimization. The experimental results on the moving MNIST dataset demonstrate that the proposed deep reweighted-RNN significantly outperforms existing RNN models.
  arXiv  Detail & Related papers  (2020-03-18T17:02:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.