Backpropagation Through Time For Networks With Long-Term Dependencies

• URL: http://arxiv.org/abs/2103.15589v1
• Date: Fri, 26 Mar 2021 15:55:54 GMT
• Title: Backpropagation Through Time For Networks With Long-Term Dependencies
• Authors: George Bird, Maxim E. Polivoda
• Abstract summary: Backpropagation through time (BPTT) is a technique of updating tuned parameters within recurrent neural networks (RNNs) We propose using the 'discrete forward sensitivity equation' and a variant of it for single and multiple interacting recurrent loops respectively. This solution is exact and also allows the network's parameters to vary between each subsequent step, however it does require the computation of a Jacobian.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Backpropagation through time (BPTT) is a technique of updating tuned parameters within recurrent neural networks (RNNs). Several attempts at creating such an algorithm have been made including: Nth Ordered Approximations and Truncated-BPTT. These methods approximate the backpropagation gradients under the assumption that the RNN only utilises short-term dependencies. This is an acceptable assumption to make for the current state of artificial neural networks. As RNNs become more advanced, a shift towards influence by long-term dependencies is likely. Thus, a new method for backpropagation is required. We propose using the 'discrete forward sensitivity equation' and a variant of it for single and multiple interacting recurrent loops respectively. This solution is exact and also allows the network's parameters to vary between each subsequent step, however it does require the computation of a Jacobian.

Related papers

• A predictive physics-aware hybrid reduced order model for reacting flows [65.73506571113623]
  A new hybrid predictive Reduced Order Model (ROM) is proposed to solve reacting flow problems. The number of degrees of freedom is reduced from thousands of temporal points to a few POD modes with their corresponding temporal coefficients. Two different deep learning architectures have been tested to predict the temporal coefficients.
  arXiv  Detail & Related papers  (2023-01-24T08:39:20Z)
• Timing-Based Backpropagation in Spiking Neural Networks Without Single-Spike Restrictions [2.8360662552057323]
  We propose a novel backpropagation algorithm for training spiking neural networks (SNNs) It encodes information in the relative multiple spike timing of individual neurons without single-spike restrictions.
  arXiv  Detail & Related papers  (2022-11-29T11:38:33Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Training Overparametrized Neural Networks in Sublinear Time [14.918404733024332]
  Deep learning comes at a tremendous computational and energy cost. We present a new and a subset of binary neural networks, as a small subset of search trees, where each corresponds to a subset of search trees (Ds) We believe this view would have further applications in analysis analysis of deep networks (Ds)
  arXiv  Detail & Related papers  (2022-08-09T02:29:42Z)
• A Taxonomy of Recurrent Learning Rules [1.4186974630564675]
  Backpropagation through time (BPTT) is the de facto standard for training recurrent neural networks (RNNs) E-prop was proposed as a causal, local, and efficient practical alternative to these algorithms. We derive RTRL from BPTT using a detailed notation bringing intuition and clarification to how they are connected.
  arXiv  Detail & Related papers  (2022-07-23T07:03:42Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks [124.33353902111939]
  We propose ADOM - an accelerated method for smooth and strongly convex decentralized optimization over time-varying networks. Up to a constant factor, which depends on the network structure only, its communication is the same as that of accelerated Nesterov method.
  arXiv  Detail & Related papers  (2021-02-18T09:37:20Z)
• Multi-Time-Scale Input Approaches for Hourly-Scale Rainfall-Runoff Modeling based on Recurrent Neural Networks [0.0]
  Two approaches are proposed to reduce the required computational time for time-series modeling through a recurrent neural network (RNN) One approach provides coarse fine temporal resolutions of the input time-series to RNN in parallel. The results confirm that both of the proposed approaches can reduce the computational time for the training of RNN significantly.
  arXiv  Detail & Related papers  (2021-01-30T07:51:55Z)
• Delay Differential Neural Networks [0.2538209532048866]
  We propose a novel model, delay differential neural networks (DDNN), inspired by delay differential equations (DDEs) For training DDNNs, we provide a memory-efficient adjoint method for computing gradients and back-propagate through the network. Experiments conducted on synthetic and real-world image classification datasets such as Cifar10 and Cifar100 show the effectiveness of the proposed models.
  arXiv  Detail & Related papers  (2020-12-12T12:20:54Z)
• A Deep-Unfolded Reference-Based RPCA Network For Video Foreground-Background Separation [86.35434065681925]
  This paper proposes a new deep-unfolding-based network design for the problem of Robust Principal Component Analysis (RPCA) Unlike existing designs, our approach focuses on modeling the temporal correlation between the sparse representations of consecutive video frames. Experimentation using the moving MNIST dataset shows that the proposed network outperforms a recently proposed state-of-the-art RPCA network in the task of video foreground-background separation.
  arXiv  Detail & Related papers  (2020-10-02T11:40:09Z)
• Belief Propagation Neural Networks [103.97004780313105]
  We introduce belief propagation neural networks (BPNNs) BPNNs operate on factor graphs and generalize Belief propagation (BP) We show that BPNNs converges 1.7x faster on Ising models while providing tighter bounds. On challenging model counting problems, BPNNs compute estimates 100's of times faster than state-of-the-art handcrafted methods.
  arXiv  Detail & Related papers  (2020-07-01T07:39:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.