SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks

• URL: http://arxiv.org/abs/2606.12287v1
• Date: Wed, 10 Jun 2026 16:23:37 GMT
• Title: SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks
• Abstract summary: Spiking Neural Networks (SNNs) are an energy-efficient alternative to conventional Artificial Neural Networks (ANNs)<n>In this paper, we propose SpikeDecoder, a fully SNN-based implementation of the Transformer decoder block.
• Score: 31.286863918611605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Transformer architecture is widely regarded as the most powerful tool for natural language processing, but due to a high number of complex operations, it inherently faces the issue of high energy consumption. To address this issue, we consider Spiking Neural Networks (SNNs), which are an energy-efficient alternative to conventional Artificial Neural Networks (ANNs) due to their naturally event-driven approach to processing information. However, this inherently makes them difficult to train. Often, many SNN-based models circumvent this issue by converting pre-trained ANNs. More recently, attempts have been made to design directly trainable SNN-based adaptations of the Transformer model structure. Although the results showed great promise, the application field was computer vision. Moreover, the proposed model incorporates only encoder blocks. In this paper, we propose SpikeDecoder, a fully SNN-based implementation of the Transformer decoder block, for applications in natural language processing. In a series of experiments, we analyze the impact of exchanging different blocks of the ANN model with spike-based alternatives to identify trade-offs and significant sources of performance loss. We further investigate the role of residual connections and the selection of SNN-compatible normalization techniques. Besides the work on the model architecture, we formulate and compare different embedding methods to project text data into spikes. Finally, we demonstrate that our proposed SNN-based decoder block reduces the theoretical energy consumption by 87% to 93% compared to the ANN baseline.

Related papers

• Reducing ANN-SNN Conversion Error via Residual Membrane Potential Alignment [52.910770415431244]
  Spiking Neural Networks (SNNs) serve as core architectures for neuromorphic computing.<n>ANN-SNN conversion circumvents issues by reusing well-trained ANN weights for low-latency, energy-efficient inference.<n>Existing conversion schemes suffer from severe accuracy drops at small timesteps, large inference delays and cumulative quantization errors.
  arXiv  Detail & Related papers  (2026-08-14T04:46:03Z)
• AS-FedBridge: Pseudo-Spike Bridge Distillation for Heterogeneous ANN-SNN Federated Learning [56.2943018570345]
  Spiking Neural Networks (SNNs) have emerged as a promising alternative to traditional Artificial Neural Networks (ANNs)<n>We propose AS-FedBridge, a novel federated learning framework tailored for mixed ANN-SNN clients.
  arXiv  Detail & Related papers  (2026-08-04T08:32:30Z)
• Large Language Models Inference Engines based on Spiking Neural Networks [5.529385616266398]
  We explore spiking neural networks (SNNs) to design transformer models.<n>A challenge in training large-scale SNNs is inefficient and time-consuming.<n>We propose NeurTransformer, a methodology for designing transformer-based SNN for inference.
  arXiv  Detail & Related papers  (2025-09-30T18:11:13Z)
• Spiking Neural Network Decision Feedback Equalization [70.3497683558609]
  We propose an SNN-based equalizer with a feedback structure akin to the decision feedback equalizer (DFE) We show that our approach clearly outperforms conventional linear equalizers for three different exemplary channels. The proposed SNN with a decision feedback structure enables the path to competitive energy-efficient transceivers.
  arXiv  Detail & Related papers  (2022-11-09T09:19:15Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision [64.71260357476602]
  Event-based vision sensors encode local pixel-wise brightness changes in streams of events rather than image frames. Recent progress in object recognition from event-based sensors has come from conversions of deep neural networks. We propose a hybrid architecture for end-to-end training of deep neural networks for event-based pattern recognition and object detection.
  arXiv  Detail & Related papers  (2021-12-06T23:45:58Z)
• Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks [1.101002667958165]
  Spiking neural networks (SNNs) have become an interesting alternative to conventional artificial neural networks (ANNs) We present a study on the key components of modern spiking architectures. We design a spiking version of the successful residual network (ResNet) architecture and test different components and training strategies on it.
  arXiv  Detail & Related papers  (2021-11-10T21:29:19Z)
• Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks [0.0]
  Spiking neural networks (SNNs) are biology-inspired artificial neural networks (ANNs) We propose a novel strategic pipeline that transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. Our method is promising to get implanted onto embedded platforms with better support of SNNs with limited energy and memory.
  arXiv  Detail & Related papers  (2021-02-28T12:04:22Z)
• Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification [4.805959718658541]
  On-device Deep Neural Networks (DNNs) have recently gained more attention due to the increasing computing power of mobile devices and the number of applications in Computer Vision (CV) and Natural Language Processing (NLP) In NLP applications, although model compression has seen initial success, there are at least three major challenges yet to be addressed: adversarial robustness, explainability, and personalization. Here we attempt to tackle these challenges by designing a new training scheme for model compression and adversarial robustness, including the optimization of an explainable feature mapping objective. The resulting compressed model is personalized using on-device private training data via fine-
  arXiv  Detail & Related papers  (2021-01-10T15:06:55Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.