Prospective Messaging: Learning in Networks with Communication Delays

• URL: http://arxiv.org/abs/2407.05494v2
• Date: Tue, 9 Jul 2024 01:20:32 GMT
• Title: Prospective Messaging: Learning in Networks with Communication Delays
• Authors: Ryan Fayyazi, Christian Weilbach, Frank Wood,
• Abstract summary: Inter-neuron communication delays are ubiquitous in physically realized neural networks. We show that delays prevent state-of-the-art continuous-time neural networks from learning even simple tasks. We then propose to compensate for communication delays by predicting future signals based on currently available ones.
• Score: 12.63723517446906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inter-neuron communication delays are ubiquitous in physically realized neural networks such as biological neural circuits and neuromorphic hardware. These delays have significant and often disruptive consequences on network dynamics during training and inference. It is therefore essential that communication delays be accounted for, both in computational models of biological neural networks and in large-scale neuromorphic systems. Nonetheless, communication delays have yet to be comprehensively addressed in either domain. In this paper, we first show that delays prevent state-of-the-art continuous-time neural networks called Latent Equilibrium (LE) networks from learning even simple tasks despite significant overparameterization. We then propose to compensate for communication delays by predicting future signals based on currently available ones. This conceptually straightforward approach, which we call prospective messaging (PM), uses only neuron-local information, and is flexible in terms of memory and computation requirements. We demonstrate that incorporating PM into delayed LE networks prevents reaction lags, and facilitates successful learning on Fourier synthesis and autoregressive video prediction tasks.

Related papers

• Contrastive Learning in Memristor-based Neuromorphic Systems [55.11642177631929]
  Spiking neural networks have become an important family of neuron-based models that sidestep many of the key limitations facing modern-day backpropagation-trained deep networks. In this work, we design and investigate a proof-of-concept instantiation of contrastive-signal-dependent plasticity (CSDP), a neuromorphic form of forward-forward-based, backpropagation-free learning.
  arXiv  Detail & Related papers  (2024-09-17T04:48:45Z)
• Expanding memory in recurrent spiking networks [2.8237889121096034]
  Recurrent spiking neural networks (RSNNs) are notoriously difficult to train because of the vanishing gradient problem that is enhanced by the binary nature of the spikes. We present a novel spiking neural network that circumvents these limitations.
  arXiv  Detail & Related papers  (2023-10-29T16:46:26Z)
• NeuralFastLAS: Fast Logic-Based Learning from Raw Data [54.938128496934695]
  Symbolic rule learners generate interpretable solutions, however they require the input to be encoded symbolically. Neuro-symbolic approaches overcome this issue by mapping raw data to latent symbolic concepts using a neural network. We introduce NeuralFastLAS, a scalable and fast end-to-end approach that trains a neural network jointly with a symbolic learner.
  arXiv  Detail & Related papers  (2023-10-08T12:33:42Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Interneurons accelerate learning dynamics in recurrent neural networks for statistical adaptation [39.245842636392865]
  We study the benefits of mediating recurrent communication via interneurons compared with direct recurrent connections. Our results suggest interneurons are useful for rapid adaptation to changing input statistics.
  arXiv  Detail & Related papers  (2022-09-21T20:03:58Z)
• Neuro-Symbolic Artificial Intelligence (AI) for Intent based Semantic Communication [85.06664206117088]
  6G networks must consider semantics and effectiveness (at end-user) of the data transmission. NeSy AI is proposed as a pillar for learning causal structure behind the observed data. GFlowNet is leveraged for the first time in a wireless system to learn the probabilistic structure which generates the data.
  arXiv  Detail & Related papers  (2022-05-22T07:11:57Z)
• Axonal Delay As a Short-Term Memory for Feed Forward Deep Spiking Neural Networks [3.985532502580783]
  Recent studies have found that the time delay of neurons plays an important role in the learning process. configuring the precise timing of the spike is a promising direction for understanding and improving the transmission process of temporal information in SNNs. In this paper, we verify the effectiveness of integrating time delay into supervised learning and propose a module that modulates the axonal delay through short-term memory.
  arXiv  Detail & Related papers  (2022-04-20T16:56:42Z)
• Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons [0.7340017786387767]
  We introduce Latent Equilibrium, a new framework for inference and learning in networks of slow components. We derive disentangled neuron and synapse dynamics from a prospective energy function. We show how our principle can be applied to detailed models of cortical microcircuitry.
  arXiv  Detail & Related papers  (2021-10-27T16:15:55Z)
• A Deep 2-Dimensional Dynamical Spiking Neuronal Network for Temporal Encoding trained with STDP [10.982390333064536]
  We show that a large, deep layered SNN with dynamical, chaotic activity mimicking the mammalian cortex is capable of encoding information from temporal data. We argue that the randomness inherent in the network weights allow the neurons to form groups that encode the temporal data being inputted after self-organizing with STDP. We analyze the network in terms of network entropy as a metric of information transfer.
  arXiv  Detail & Related papers  (2020-09-01T17:12:18Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• ResiliNet: Failure-Resilient Inference in Distributed Neural Networks [56.255913459850674]
  We introduce ResiliNet, a scheme for making inference in distributed neural networks resilient to physical node failures. Failout simulates physical node failure conditions during training using dropout, and is specifically designed to improve the resiliency of distributed neural networks.
  arXiv  Detail & Related papers  (2020-02-18T05:58:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.