Spike-driven Large Language Model

• URL: http://arxiv.org/abs/2604.16475v1
• Date: Sat, 11 Apr 2026 17:58:35 GMT
• Title: Spike-driven Large Language Model
• Abstract summary: Spiking Neural Networks (SNNs) possess spike-driven characteristics.<n>Current Large Language Models (LLMs) are primarily based on large-scale dense matrix multiplications.<n>We propose SDLLM, a spike-driven large language model that eliminates dense matrix multiplications through sparse addition operations.
• Score: 49.52947423982696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current Large Language Models (LLMs) are primarily based on large-scale dense matrix multiplications. Inspired by the brain's information processing mechanism, we explore the fundamental question: how to effectively integrate the brain's spiking-driven characteristics into LLM inference. Spiking Neural Networks (SNNs) possess spike-driven characteristics, and some works have attempted to combine SNNs with Transformers. However, achieving spike-driven LLMs with billions of parameters, relying solely on sparse additions, remains a challenge in the SNN field. To address the issues of limited representational capacity and sparsity in existing spike encoding schemes at the LLM level, we propose SDLLM, a spike-driven large language model that eliminates dense matrix multiplications through sparse addition operations. Specifically, we use the plug-and-play gamma-SQP two-step spike encoding method to ensure that the quantization process aligns with the model's semantic space, mitigating representation degradation caused by binary spikes. Furthermore, we introduce bidirectional encoding under symmetric quantization and membrane potential clipping mechanisms, leading to spike trains with no or low firing counts dominating, significantly reducing the model's spike firing rate, while halving the number of time steps. Experimental results show that SDLLM not only significantly reduces inference costs but also achieves state-of-the-art task performance under the spike-based paradigm. For example, compared to previous spike-based LLMs, SDLLM reduces energy consumption by 7x and improves accuracy by 4.2%. Our model provides inspiration for the architecture design of the next generation of event-driven neuromorphic chips.

Related papers

• C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs [23.275426637407033]
  Multimodal large language models (MLLMs) require huge memory and computational costs.<n>Post-training quantization (PTQ) techniques offer an efficient solution for model compression and inference acceleration.<n>We propose C-PTQ, a unified channel-wise PTQ method that harmonizes task-specific loss perturbation and quantization error.
  arXiv  Detail & Related papers  (2026-07-23T09:08:57Z)
• REALM: Retrospective Encoder Alignment for LFP Modeling [2.8036525859094823]
  Spike activity has been the dominant neural signal for behavior decoding due to its high spatial and temporal resolution.<n>As brain-computer interfaces (BCIs) move toward high channel counts and wireless operation, the high sampling frequency of spike signals becomes a bottleneck.<n>We propose REALM: a retrospective distillation framework that enables causal LFP decoding.
  arXiv  Detail & Related papers  (2026-05-14T14:16:22Z)
• On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks [0.09665867043802821]
  Auto-regressive Large Language Models (LLMs) achieve strong performance on coding tasks, but incur high memory and inference costs.<n>We investigate the application and robustness of PTQ techniques, specifically GPTQ and a modified Hessian-Aware Quantization (HAWQ) algorithm.
  arXiv  Detail & Related papers  (2026-04-22T00:53:43Z)
• Neural Dynamics Self-Attention for Spiking Transformers [59.57278823240336]
  Spiking Neural Networks (SNNs) with Transformer architectures offer a promising pathway to balance energy efficiency and performance.<n>Existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference.<n>We propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements.
  arXiv  Detail & Related papers  (2026-03-09T12:23:37Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processing [38.632763514823864]
  IML-Spikeformer is a spiking Transformer architecture specifically designed for large-scale speech processing.<n>IML-Spikeformer achieves word error rates of 6.0% on AiShell-1 and 3.4% on Libri-960.
  arXiv  Detail & Related papers  (2025-07-10T03:26:24Z)
• Pushing the Limits of Large Language Model Quantization via the Linearity Theorem [71.3332971315821]
  We present a "line theoremarity" establishing a direct relationship between the layer-wise $ell$ reconstruction error and the model perplexity increase due to quantization. This insight enables two novel applications: (1) a simple data-free LLM quantization method using Hadamard rotations and MSE-optimal grids, dubbed HIGGS, and (2) an optimal solution to the problem of finding non-uniform per-layer quantization levels.
  arXiv  Detail & Related papers  (2024-11-26T15:35:44Z)
• SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking [43.275370104552344]
  The human brain is much more energy-efficient than large language models with similar parameters.<n>We propose the first spiking large language model, SpikeLLM.<n>SpikeLLM reduces 11.01% WikiText2 perplexity and improves 2.55% accuracy of common scene reasoning.
  arXiv  Detail & Related papers  (2024-07-05T08:37:17Z)
• Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment [56.44025052765861]
  Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs. We show a total speedup on CPUs for sparse-quantized LLaMA models of up to 8.6x.
  arXiv  Detail & Related papers  (2024-05-06T16:03:32Z)
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition [67.95996816744251]
  State-of-the-art language models (LMs) represented by long-short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming increasingly complex and expensive for practical applications. Current quantization methods are based on uniform precision and fail to account for the varying performance sensitivity at different parts of LMs to quantization errors. Novel mixed precision neural network LM quantization methods are proposed in this paper.
  arXiv  Detail & Related papers  (2021-11-29T12:24:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.