Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsity

• URL: http://arxiv.org/abs/2404.00880v1
• Date: Mon, 1 Apr 2024 03:18:42 GMT
• Title: Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsity
• Authors: Quincy Hershey, Randy Paffenroth, Harsh Pathak, Simon Tavener,
• Abstract summary: We show that many common neural network models, such as Recurrent Neural Networks (RNN), can all be represented as iterative maps. RNNs are known to be Turing complete, and therefore capable of representing any computable function. This perspective leads to several insights that illuminate both theoretical and practical aspects of NNs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks (NN) can be divided into two broad categories, recurrent and non-recurrent. Both types of neural networks are popular and extensively studied, but they are often treated as distinct families of machine learning algorithms. In this position paper, we argue that there is a closer relationship between these two types of neural networks than is normally appreciated. We show that many common neural network models, such as Recurrent Neural Networks (RNN), Multi-Layer Perceptrons (MLP), and even deep multi-layer transformers, can all be represented as iterative maps. The close relationship between RNNs and other types of NNs should not be surprising. In particular, RNNs are known to be Turing complete, and therefore capable of representing any computable function (such as any other types of NNs), but herein we argue that the relationship runs deeper and is more practical than this. For example, RNNs are often thought to be more difficult to train than other types of NNs, with RNNs being plagued by issues such as vanishing or exploding gradients. However, as we demonstrate in this paper, MLPs, RNNs, and many other NNs lie on a continuum, and this perspective leads to several insights that illuminate both theoretical and practical aspects of NNs.

Related papers

• Investigating Sparsity in Recurrent Neural Networks [0.0]
  This thesis focuses on investigating the effects of pruning and Sparse Recurrent Neural Networks on the performance of RNNs. We first describe the pruning of RNNs, its impact on the performance of RNNs, and the number of training epochs required to regain accuracy after the pruning is performed. Next, we continue with the creation and training of Sparse Recurrent Neural Networks and identify the relation between the performance and the graph properties of its underlying arbitrary structure.
  arXiv  Detail & Related papers  (2024-07-30T07:24:58Z)
• Tensor Networks Meet Neural Networks: A Survey and Future Perspectives [27.878669143107885]
  tensorial neural networks (TNNs) and neural networks (NNs) are two fundamental data modeling approaches. TNs solve the curse of dimensionality in large-scale tensors by converting an exponential number of dimensions to complexity. NNs have displayed exceptional performance in various applications, e.g., computer vision, natural language processing, and robotics research.
  arXiv  Detail & Related papers  (2023-01-22T17:35:56Z)
• Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study [55.12108376616355]
  The study on NTK has been devoted to typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp) In this work, we derive the finite-width-K formulation for a special class of NNs-Hp, i.e., neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK.
  arXiv  Detail & Related papers  (2022-09-16T06:36:06Z)
• Linear Leaky-Integrate-and-Fire Neuron Model Based Spiking Neural Networks and Its Mapping Relationship to Deep Neural Networks [7.840247953745616]
  Spiking neural networks (SNNs) are brain-inspired machine learning algorithms with merits such as biological plausibility and unsupervised learning capability. This paper establishes a precise mathematical mapping between the biological parameters of the Linear Leaky-Integrate-and-Fire model (LIF)/SNNs and the parameters of ReLU-AN/Deep Neural Networks (DNNs)
  arXiv  Detail & Related papers  (2022-05-31T17:02:26Z)
• Redundant representations help generalization in wide neural networks [71.38860635025907]
  We study the last hidden layer representations of various state-of-the-art convolutional neural networks. We find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
  arXiv  Detail & Related papers  (2021-06-07T10:18:54Z)
• A Practical Tutorial on Graph Neural Networks [49.919443059032226]
  Graph neural networks (GNNs) have recently grown in popularity in the field of artificial intelligence (AI) This tutorial exposes the power and novelty of GNNs to AI practitioners.
  arXiv  Detail & Related papers  (2020-10-11T12:36:17Z)
• Exploiting Heterogeneity in Operational Neural Networks by Synaptic Plasticity [87.32169414230822]
  Recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional Neural Networks (CNNs) In this study the focus is drawn on searching the best-possible operator set(s) for the hidden neurons of the network based on the Synaptic Plasticity paradigm that poses the essential learning theory in biological neurons. Experimental results over highly challenging problems demonstrate that the elite ONNs even with few neurons and layers can achieve a superior learning performance than GIS-based ONNs.
  arXiv  Detail & Related papers  (2020-08-21T19:03:23Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Neural Additive Models: Interpretable Machine Learning with Neural Nets [77.66871378302774]
  Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn a linear combination of neural networks that each attend to a single input feature.
  arXiv  Detail & Related papers  (2020-04-29T01:28:32Z)
• ResNets, NeuralODEs and CT-RNNs are Particular Neural Regulatory Networks [10.518340300810504]
  This paper shows that ResNets, NeuralODEs, and CT-RNNs, are particular neural regulatory networks (NRNs) NRNs are a biophysical model for the nonspiking neurons encountered in small species, such as the C.elegans nematode, and in the retina of large species. For a given approximation task, this considerable succinctness allows to learn a very small and therefore understandable NRN.
  arXiv  Detail & Related papers  (2020-02-26T18:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.