Learning Long-Range Dependencies with Temporal Predictive Coding

• URL: http://arxiv.org/abs/2602.18131v1
• Date: Fri, 20 Feb 2026 10:46:28 GMT
• Title: Learning Long-Range Dependencies with Temporal Predictive Coding
• Authors: Tom Potter, Oliver Rhodes,
• Abstract summary: This work introduces a novel method combining Temporal Predictive Coding (tPC) approximate with Real-Time Recurrent Learning (RLRL)<n>Results indicate that the proposed method can closely match the performance of BPTT on both synthetic benchmarks and real-world tasks.<n>On a challenging machine translation task, with a 15-million parameter model, the proposed method achieves a test perplexity of 7.62 (vs. 7.49 for BPTT)
• Score: 0.31401665995867667
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predictive Coding (PC) is a biologically-inspired learning framework characterised by local, parallelisable operations, properties that enable energy-efficient implementation on neuromorphic hardware. Despite this, extending PC effectively to recurrent neural networks (RNNs) has been challenging, particularly for tasks involving long-range temporal dependencies. Backpropagation Through Time (BPTT) remains the dominant method for training RNNs, but its non-local computation, lack of spatial parallelism, and requirement to store extensive activation histories results in significant energy consumption. This work introduces a novel method combining Temporal Predictive Coding (tPC) with approximate Real-Time Recurrent Learning (RTRL), enabling effective spatio-temporal credit assignment. Results indicate that the proposed method can closely match the performance of BPTT on both synthetic benchmarks and real-world tasks. On a challenging machine translation task, with a 15-million parameter model, the proposed method achieves a test perplexity of 7.62 (vs. 7.49 for BPTT), marking one of the first applications of tPC to tasks of this scale. These findings demonstrate the potential of this method to learn complex temporal dependencies whilst retaining the local, parallelisable, and flexible properties of the original PC framework, paving the way for more energy-efficient learning systems.

Related papers

• When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training [58.25341036646294]
  We analytically examine why learning recurrent poles does not provide tangible benefits in data and empirically offer real-time learning scenarios.<n>We show that fixed-pole networks achieve superior performance with lower training complexity, making them more suitable for online real-time tasks.
  arXiv  Detail & Related papers  (2026-02-25T00:15:13Z)
• Sample-Efficient Neurosymbolic Deep Reinforcement Learning [49.60927398960061]
  We propose a neuro-symbolic Deep RL approach that integrates background symbolic knowledge to improve sample efficiency.<n>Online reasoning is performed to guide the training process through two mechanisms.<n>We show improved performance over a state-of-the-art reward machine baseline.
  arXiv  Detail & Related papers  (2026-01-06T09:28:53Z)
• Fast Training of Recurrent Neural Networks with Stationary State Feedbacks [48.22082789438538]
  Recurrent neural networks (RNNs) have recently demonstrated strong performance and faster inference than Transformers.<n>We propose a novel method that replaces BPTT with a fixed gradient feedback mechanism.
  arXiv  Detail & Related papers  (2025-03-29T14:45:52Z)
• TESS: A Scalable Temporally and Spatially Local Learning Rule for Spiking Neural Networks [6.805933498669221]
  Training neural networks (SNNs) on resource-constrained devices remains challenging due to high computational and memory demands.<n>We introduce TESS, a temporally and spatially local learning rule for training SNNs.<n>Our approach addresses both temporal and spatial credit assignments by relying solely on locally available signals within each neuron.
  arXiv  Detail & Related papers  (2025-02-03T21:23:15Z)
• LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization [6.738409533239947]
  Training deep neural networks (DNNs) using traditional backpropagation (BP) presents challenges in terms of computational complexity and energy consumption. We propose a novel Local Learning rule inspired by neural activity Synchronization phenomena (LLS) observed in the brain. LLS achieves comparable performance with up to $300 times$ fewer multiply-accumulate (MAC) operations and half the memory requirements of BP.
  arXiv  Detail & Related papers  (2024-05-24T18:24:24Z)
• Real-Time Recurrent Reinforcement Learning [7.737685867200335]
  We introduce a biologically plausible RL framework for solving tasks in partially observable Markov decision processes (POMDPs)<n>The proposed algorithm combines three integral parts: (1) A Meta-RL architecture, resembling the mammalian basal ganglia; (2) A biologically plausible reinforcement learning algorithm, exploiting temporal difference learning and eligibility traces to train the policy and the value-function; and (3) An online automatic differentiation algorithm for computing the gradients with respect to parameters of a shared recurrent network backbone.
  arXiv  Detail & Related papers  (2023-11-08T16:56:16Z)
• TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling [54.97005925277638]
  The identification of sensory cues associated with potential opportunities and dangers is frequently complicated by unrelated events that separate useful cues by long delays. It remains a challenging task for state-of-the-art spiking neural networks (SNNs) to establish long-term temporal dependency between distant cues. We propose a novel biologically inspired Two-Compartment Leaky Integrate-and-Fire spiking neuron model, dubbed TC-LIF.
  arXiv  Detail & Related papers  (2023-08-25T08:54:41Z)
• S-TLLR: STDP-inspired Temporal Local Learning Rule for Spiking Neural Networks [7.573297026523597]
  Spiking Neural Networks (SNNs) are biologically plausible models that have been identified as potentially apt for deploying energy-efficient intelligence at the edge. We propose S-TLLR, a novel three-factor temporal local learning rule inspired by the Spike-Timing Dependent Plasticity (STDP) mechanism. S-TLLR is designed to have low memory and time complexities, which are independent of the number of time steps, rendering it suitable for online learning on low-power edge devices.
  arXiv  Detail & Related papers  (2023-06-27T05:44:56Z)
• Efficient Real Time Recurrent Learning through combined activity and parameter sparsity [0.5076419064097732]
  Backpropagation through time (BPTT) is the standard algorithm for training recurrent neural networks (RNNs) BPTT is unsuited for online learning and presents a challenge for implementation on low-resource real-time systems. We show that recurrent networks exhibiting high activity sparsity can reduce the computational cost of Real-Time Recurrent Learning (RTRL)
  arXiv  Detail & Related papers  (2023-03-10T01:09:04Z)
• ETLP: Event-based Three-factor Local Plasticity for online learning with neuromorphic hardware [105.54048699217668]
  We show a competitive performance in accuracy with a clear advantage in the computational complexity for Event-Based Three-factor Local Plasticity (ETLP) We also show that when using local plasticity, threshold adaptation in spiking neurons and a recurrent topology are necessary to learntemporal patterns with a rich temporal structure.
  arXiv  Detail & Related papers  (2023-01-19T19:45:42Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Deep Q-network using reservoir computing with multi-layered readout [0.0]
  Recurrent neural network (RNN) based reinforcement learning (RL) is used for learning context-dependent tasks. An approach with replay memory introducing reservoir computing has been proposed, which trains an agent without BPTT. This paper shows that the performance of this method improves by using a multi-layered neural network for the readout layer.
  arXiv  Detail & Related papers  (2022-03-03T00:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.