Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks

• URL: http://arxiv.org/abs/2111.12963v1
• Date: Thu, 25 Nov 2021 08:14:55 GMT
• Title: Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks
• Authors: Tilahun M. Getu
• Abstract summary: Theory of deep learning has spurred the theory of deep learning-oriented depth and breadth of developments. Motivated by such developments, we pose fundamental questions: can we accurately approximate an arbitrary matrix-vector product using deep rectified linear unit (ReLU) feedforward neural networks (FNNs)? We derive error bounds in Lebesgue and Sobolev norms that comprise our developed deep approximation theory. The developed theory is also applicable for guiding and easing the training of teacher deep ReLU FNNs in view of the emerging teacher-student AI or ML paradigms.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Among the several paradigms of artificial intelligence (AI) or machine learning (ML), a remarkably successful paradigm is deep learning. Deep learning's phenomenal success has been hoped to be interpreted via fundamental research on the theory of deep learning. Accordingly, applied research on deep learning has spurred the theory of deep learning-oriented depth and breadth of developments. Inspired by such developments, we pose these fundamental questions: can we accurately approximate an arbitrary matrix-vector product using deep rectified linear unit (ReLU) feedforward neural networks (FNNs)? If so, can we bound the resulting approximation error? In light of these questions, we derive error bounds in Lebesgue and Sobolev norms that comprise our developed deep approximation theory. Guided by this theory, we have successfully trained deep ReLU FNNs whose test results justify our developed theory. The developed theory is also applicable for guiding and easing the training of teacher deep ReLU FNNs in view of the emerging teacher-student AI or ML paradigms that are essential for solving several AI or ML problems in wireless communications and signal processing; network science and graph signal processing; and network neuroscience and brain physics.

Related papers

• Deep learning applied to computational mechanics: A comprehensive review, state of the art, and the classics [77.34726150561087]
  Recent developments in artificial neural networks, particularly deep learning (DL), are reviewed in detail. Both hybrid and pure machine learning (ML) methods are discussed. History and limitations of AI are recounted and discussed, with particular attention at pointing out misstatements or misconceptions of the classics.
  arXiv  Detail & Related papers  (2022-12-18T02:03:00Z)
• Information Flow in Deep Neural Networks [0.6922389632860545]
  There is no comprehensive theoretical understanding of how deep neural networks work or are structured. Deep networks are often seen as black boxes with unclear interpretations and reliability. This work aims to apply principles and techniques from information theory to deep learning models to increase our theoretical understanding and design better algorithms.
  arXiv  Detail & Related papers  (2022-02-10T23:32:26Z)
• Credit Assignment in Neural Networks through Deep Feedback Control [59.14935871979047]
  Deep Feedback Control (DFC) is a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose control signal can be used for credit assignment. The resulting learning rule is fully local in space and time and approximates Gauss-Newton optimization for a wide range of connectivity patterns. To further underline its biological plausibility, we relate DFC to a multi-compartment model of cortical pyramidal neurons with a local voltage-dependent synaptic plasticity rule, consistent with recent theories of dendritic processing.
  arXiv  Detail & Related papers  (2021-06-15T05:30:17Z)
• A Critical Review of Information Bottleneck Theory and its Applications to Deep Learning [0.0]
  Deep neural networks have seen unparalleled improvements that continue to impact every aspect of today's society. The information bottleneck theory has emerged as a promising approach to better understand the learning dynamics of neural networks. The goal of this survey is to provide a comprehensive review of IB theory covering it's information theoretic roots and the recently proposed applications to understand deep learning models.
  arXiv  Detail & Related papers  (2021-05-07T14:16:38Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• A Study of the Mathematics of Deep Learning [1.14219428942199]
  "Deep Learning"/"Deep Neural Nets" is a technological marvel that is now increasingly deployed at the cutting-edge of artificial intelligence tasks. This thesis takes several steps towards building strong theoretical foundations for these new paradigms of deep-learning.
  arXiv  Detail & Related papers  (2021-04-28T22:05:54Z)
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks [80.55378250013496]
  We study how neural networks trained by gradient descent extrapolate what they learn outside the support of the training distribution. Graph Neural Networks (GNNs) have shown some success in more complex tasks.
  arXiv  Detail & Related papers  (2020-09-24T17:48:59Z)
• Deep Q-Learning: Theoretical Insights from an Asymptotic Analysis [3.9871041399267613]
  Deep Q-Learning is an important reinforcement learning algorithm, which involves training a deep neural network to approximate the well-known Q-function. Although wildly successful under laboratory conditions, serious gaps between theory and practice as well as a lack of formal guarantees prevent its use in the real world. We provide a theoretical analysis of a popular version of Deep Q-Learning under realistic verifiable assumptions.
  arXiv  Detail & Related papers  (2020-08-25T07:59:20Z)
• A Theoretical Framework for Target Propagation [75.52598682467817]
  We analyze target propagation (TP), a popular but not yet fully understood alternative to backpropagation (BP) Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training.
  arXiv  Detail & Related papers  (2020-06-25T12:07:06Z)
• Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G Networks [84.2155885234293]
  We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC. To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
  arXiv  Detail & Related papers  (2020-02-22T14:38:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.