NeuronSpark: A Spiking Neural Network Language Model with Selective State Space Dynamics

• URL: http://arxiv.org/abs/2603.16148v1
• Date: Tue, 17 Mar 2026 06:01:05 GMT
• Title: NeuronSpark: A Spiking Neural Network Language Model with Selective State Space Dynamics
• Abstract summary: We introduce NeuronSpark, a 0.9B- parameter SNN language model trained with next-token prediction and surrogate gradients.<n>Under a constrained budget, NeuronSpark-0.9B reaches 3.6 pretraining loss and shows early multi-turn dialogue behavior after SFT.<n>These results support the feasibility of end-to-end language modeling with a pure SNN architecture at this scale.
• Score: 0.7614628596146601
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We ask whether a pure spiking backbone can learn large-scale language modeling from random initialization, without Transformer distillation. We introduce NeuronSpark, a 0.9B-parameter SNN language model trained with next-token prediction and surrogate gradients. The model combines selective state-space spiking dynamics, leakage-current inter-layer communication, PonderNet adaptive timesteps, fused Triton PLIF kernels, and stabilization techniques (residual centering, lateral-inhibition normalization, and natural-gradient compensation). Under a constrained budget (about 1.4B pretraining tokens and 6.5K SFT steps), NeuronSpark-0.9B reaches 3.6 pretraining loss and shows early multi-turn dialogue behavior after SFT. These results support the feasibility of end-to-end language modeling with a pure SNN architecture at this scale.

Related papers

• URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretraining [1.185706452828542]
  The BabyLM challenge measures how much language a model can learn from developmentally-plausible, child-scale data.<n>This paper presents URCHIN, which applies the Parallelized Hierarchical Connectome Spiking State-space Model to language modeling.<n>Across all three BabyLM tracks (Strict-100M, Strict-Small, and Multilingual), URCHIN offers a biologically plausible, efficient, and directly deployable reference point.
  arXiv  Detail & Related papers  (2026-09-12T12:06:01Z)
• PTS-SNN: A Prompt-Tuned Temporal Shift Spiking Neural Networks for Efficient Speech Emotion Recognition [12.087823767638788]
  Speech Emotion Recognition (SER) is widely deployed in Human-Computer Interaction, yet the high computational cost hinders their implementation on resource-constrained edge devices.<n>We propose Prompt-Tuned Spiking Neural Networks (PTS-SNN), a parameter-efficient neuromorphic adaptation with spiking dynamics.
  arXiv  Detail & Related papers  (2026-02-09T03:29:16Z)
• Discretized Quadratic Integrate-and-Fire Neuron Model for Deep Spiking Neural Networks [0.08749675983608168]
  Spiking Neural Networks (SNNs) have emerged as energy-efficient alternatives to traditional artificial neural networks.<n>We propose the first discretization of the Quadratic Integrate-and-Fire (QIF) neuron model tailored for high-performance deep spiking neural networks.
  arXiv  Detail & Related papers  (2025-10-05T02:30:10Z)
• Langevin Flows for Modeling Neural Latent Dynamics [81.81271685018284]
  We introduce LangevinFlow, a sequential Variational Auto-Encoder where the time evolution of latent variables is governed by the underdamped Langevin equation.<n>Our approach incorporates physical priors -- such as inertia, damping, a learned potential function, and forces -- to represent both autonomous and non-autonomous processes in neural systems.<n>Our method outperforms state-of-the-art baselines on synthetic neural populations generated by a Lorenz attractor.
  arXiv  Detail & Related papers  (2025-07-15T17:57:48Z)
• Initial Model Incorporation for Deep Learning FWI: Pretraining or Denormalization? [7.032076746858836]
  Subsurface property neural network reparameterized full waveform inversion (FWI) has emerged as an effective unsupervised learning framework.<n>It updates the trainable neural network parameters instead of fine-tuning on the subsurface model directly.<n>Pretraining first regulates the neural networks' parameters by fitting the initial velocity model.<n>Denormalization directly adds the outputs of the network into the initial models without pretraining.
  arXiv  Detail & Related papers  (2025-06-05T18:06:37Z)
• Forecasting the 2016-2017 Central Apennines Earthquake Sequence with a Neural Point Process [0.0]
  We investigate whether flexible point process models can be applied to short-term seismicity forecasting. We show how a temporal neural model can forecast earthquakes above a target magnitude threshold.
  arXiv  Detail & Related papers  (2023-01-24T12:15:12Z)
• Bayesian Neural Network Language Modeling for Speech Recognition [59.681758762712754]
  State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex. In this paper, an overarching full Bayesian learning framework is proposed to account for the underlying uncertainty in LSTM-RNN and Transformer LMs.
  arXiv  Detail & Related papers  (2022-08-28T17:50:19Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Dependency-based Mixture Language Models [53.152011258252315]
  We introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention.
  arXiv  Detail & Related papers  (2022-03-19T06:28:30Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Flexible Transmitter Network [84.90891046882213]
  Current neural networks are mostly built upon the MP model, which usually formulates the neuron as executing an activation function on the real-valued weighted aggregation of signals received from other neurons. We propose the Flexible Transmitter (FT) model, a novel bio-plausible neuron model with flexible synaptic plasticity. We present the Flexible Transmitter Network (FTNet), which is built on the most common fully-connected feed-forward architecture.
  arXiv  Detail & Related papers  (2020-04-08T06:55:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.