Astrocyte-Enabled Advancements in Spiking Neural Networks for Large Language Modeling

• URL: http://arxiv.org/abs/2312.07625v2
• Date: Tue, 26 Dec 2023 02:54:29 GMT
• Title: Astrocyte-Enabled Advancements in Spiking Neural Networks for Large Language Modeling
• Authors: Guobin Shen, Dongcheng Zhao, Yiting Dong, Yang Li, Jindong Li, Kang Sun, Yi Zeng
• Abstract summary: Astrocyte-Modulated Spiking Neural Network (AstroSNN) exhibits exceptional performance in tasks involving memory retention and natural language generation. AstroSNN shows low latency, high throughput, and reduced memory usage in practical applications.
• Score: 7.863029550014263
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Within the complex neuroarchitecture of the brain, astrocytes play crucial roles in development, structure, and metabolism. These cells regulate neural activity through tripartite synapses, directly impacting cognitive processes such as learning and memory. Despite the growing recognition of astrocytes' significance, traditional Spiking Neural Network (SNN) models remain predominantly neuron-centric, overlooking the profound influence of astrocytes on neural dynamics. Inspired by these biological insights, we have developed an Astrocyte-Modulated Spiking Unit (AM-SU), an innovative framework that integrates neuron-astrocyte interactions into the computational paradigm, demonstrating wide applicability across various hardware platforms. Our Astrocyte-Modulated Spiking Neural Network (AstroSNN) exhibits exceptional performance in tasks involving memory retention and natural language generation, particularly in handling long-term dependencies and complex linguistic structures. The design of AstroSNN not only enhances its biological authenticity but also introduces novel computational dynamics, enabling more effective processing of complex temporal dependencies. Furthermore, AstroSNN shows low latency, high throughput, and reduced memory usage in practical applications, making it highly suitable for resource-constrained environments. By successfully integrating astrocytic dynamics into intelligent neural networks, our work narrows the gap between biological plausibility and neural modeling, laying the groundwork for future biologically-inspired neural computing research that includes both neurons and astrocytes.

Related papers

• Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling [52.06722364186432]
  We propose a biologically-informed framework for enhancing artificial neural networks (ANNs) Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors. We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
  arXiv  Detail & Related papers  (2024-07-05T14:11:28Z)
• Astrocytes as a mechanism for meta-plasticity and contextually-guided network function [2.66269503676104]
  Astrocytes are a ubiquitous and enigmatic type of non-neuronal cell. Astrocytes may play a more direct and active role in brain function and neural computation.
  arXiv  Detail & Related papers  (2023-11-06T20:31:01Z)
• The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks [64.08042492426992]
  We introduce the Expressive Memory (ELM) neuron model, a biologically inspired model of a cortical neuron. Our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. We evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets.
  arXiv  Detail & Related papers  (2023-06-14T13:34:13Z)
• Learning to Act through Evolution of Neural Diversity in Random Neural Networks [9.387749254963595]
  In most artificial neural networks (ANNs), neural computation is abstracted to an activation function that is usually shared between all neurons. We propose the optimization of neuro-centric parameters to attain a set of diverse neurons that can perform complex computations.
  arXiv  Detail & Related papers  (2023-05-25T11:33:04Z)
• Dive into the Power of Neuronal Heterogeneity [8.6837371869842]
  We show the challenges faced by backpropagation-based methods in optimizing Spiking Neural Networks (SNNs) and achieve more robust optimization of heterogeneous neurons in random networks using an Evolutionary Strategy (ES) We find that membrane time constants play a crucial role in neural heterogeneity, and their distribution is similar to that observed in biological experiments.
  arXiv  Detail & Related papers  (2023-05-19T07:32:29Z)
• Contrastive-Signal-Dependent Plasticity: Self-Supervised Learning in Spiking Neural Circuits [61.94533459151743]
  This work addresses the challenge of designing neurobiologically-motivated schemes for adjusting the synapses of spiking networks. Our experimental simulations demonstrate a consistent advantage over other biologically-plausible approaches when training recurrent spiking networks.
  arXiv  Detail & Related papers  (2023-03-30T02:40:28Z)
• Constraints on the design of neuromorphic circuits set by the properties of neural population codes [61.15277741147157]
  In the brain, information is encoded, transmitted and used to inform behaviour. Neuromorphic circuits need to encode information in a way compatible to that used by populations of neuron in the brain.
  arXiv  Detail & Related papers  (2022-12-08T15:16:04Z)
• Towards efficient end-to-end speech recognition with biologically-inspired neural networks [10.457580011403289]
  We introduce neural connectivity concepts emulating the axo-somatic and the axo-axonic synapses. We demonstrate for the first time, that a biologically realistic implementation of a large-scale ASR model can yield competitive performance levels.
  arXiv  Detail & Related papers  (2021-10-04T21:24:10Z)
• Continuous Learning and Adaptation with Membrane Potential and Activation Threshold Homeostasis [91.3755431537592]
  This paper presents the Membrane Potential and Activation Threshold Homeostasis (MPATH) neuron model. The model allows neurons to maintain a form of dynamic equilibrium by automatically regulating their activity when presented with input. Experiments demonstrate the model's ability to adapt to and continually learn from its input.
  arXiv  Detail & Related papers  (2021-04-22T04:01:32Z)
• Recurrent Neural Network Learning of Performance and Intrinsic Population Dynamics from Sparse Neural Data [77.92736596690297]
  We introduce a novel training strategy that allows learning not only the input-output behavior of an RNN but also its internal network dynamics. We test the proposed method by training an RNN to simultaneously reproduce internal dynamics and output signals of a physiologically-inspired neural model. Remarkably, we show that the reproduction of the internal dynamics is successful even when the training algorithm relies on the activities of a small subset of neurons.
  arXiv  Detail & Related papers  (2020-05-05T14:16:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.