URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretraining

• URL: http://arxiv.org/abs/2609.13899v1
• Date: Sat, 12 Sep 2026 12:06:01 GMT
• Title: URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretraining
• Abstract summary: The BabyLM challenge measures how much language a model can learn from developmentally-plausible, child-scale data.<n>This paper presents URCHIN, which applies the Parallelized Hierarchical Connectome Spiking State-space Model to language modeling.<n>Across all three BabyLM tracks (Strict-100M, Strict-Small, and Multilingual), URCHIN offers a biologically plausible, efficient, and directly deployable reference point.
• Score: 1.185706452828542
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The BabyLM challenge measures how much language a model can learn from developmentally-plausible, child-scale data rather than internet-scale corpora, yet prior language models forgo the biological constraints of the neural circuitry that acquires human language: spiking neurons separated into excitatory and inhibitory populations wired by a recurrent lateral connectome. This paper presents URCHIN (Unified Recurrent Connectome with Horizontal Integrate-and-fire Neurons), which applies the Parallelized Hierarchical Connectome Spiking State-space Model (PHCSSM) to language modeling: leaky integrate-and-fire neurons coupled by a Dale's-law lateral connectome resolve each token through a multi-transmission loop that recirculates activity to a fixed point. The instantiation is deliberately minimal: a single horizontal layer of 128 neurons, no attention, and 4.23M parameters. Two implementations share one set of weights and produce identical benchmark scores, so URCHIN is trained once and deployed either way with no conversion step: a parallel state-space model (SSM) scan that is GPU-efficient for training, or an event-driven recurrent spiking neural network (RSNN) with constant-cost inference for CPU or neuromorphic edge deployment. Across all three BabyLM tracks (Strict-100M, Strict-Small, and Multilingual), URCHIN offers a biologically plausible, efficient, and directly deployable reference point.

Related papers

• Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons [27.855778669417592]
  Cortical neurons are complex, multi-timescale processors wired into recurrent circuits.<n>Machine learning predominantly builds models extremely simple units, a default inherited from early neural-network theory.<n>We introduce the ELM Network, whose recurrent layer is built from Expressive Memory (ELM) neurons.
  arXiv  Detail & Related papers  (2026-05-12T12:29:33Z)
• UniBCI: Towards a Unified Pretrained Model for Invasive Brain-Computer Interfaces [7.71021447675552]
  Existing approaches face critical challenges, including limited-scale heterogeneous data, cross-domain distribution shift, and fewer trainable neural signals.<n>In this work, we propose a unified pre-trained model for invasive Brain Interfaces.<n>Results suggest that UniBCI provides a practical step toward general-purpose neural foundation models.
  arXiv  Detail & Related papers  (2026-04-30T06:30:24Z)
• NeuronSpark: A Spiking Neural Network Language Model with Selective State Space Dynamics [0.7614628596146601]
  We introduce NeuronSpark, a 0.9B- parameter SNN language model trained with next-token prediction and surrogate gradients.<n>Under a constrained budget, NeuronSpark-0.9B reaches 3.6 pretraining loss and shows early multi-turn dialogue behavior after SFT.<n>These results support the feasibility of end-to-end language modeling with a pure SNN architecture at this scale.
  arXiv  Detail & Related papers  (2026-03-17T06:01:05Z)
• PTS-SNN: A Prompt-Tuned Temporal Shift Spiking Neural Networks for Efficient Speech Emotion Recognition [12.087823767638788]
  Speech Emotion Recognition (SER) is widely deployed in Human-Computer Interaction, yet the high computational cost hinders their implementation on resource-constrained edge devices.<n>We propose Prompt-Tuned Spiking Neural Networks (PTS-SNN), a parameter-efficient neuromorphic adaptation with spiking dynamics.
  arXiv  Detail & Related papers  (2026-02-09T03:29:16Z)
• Language Model Circuits Are Sparse in the Neuron Basis [50.460651620833055]
  We show that textbfMLP neurons are as sparse a feature basis as SAEs.<n>This work advances automated interpretability of language models without additional training costs.
  arXiv  Detail & Related papers  (2026-01-30T05:41:19Z)
• A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks [30.855279392147082]
  Spikachu is a scalable, causal, and energy-efficient neural decoding framework based on SNNs.<n>We evaluate our approach on 113 recording sessions from 6 non-human primates.<n>Our method outperforms causal baselines when trained on single sessions using between 2.26 and 418.81 times less energy.
  arXiv  Detail & Related papers  (2025-10-23T15:55:45Z)
• In-Context Language Learning: Architectures and Algorithms [73.93205821154605]
  We study ICL through the lens of a new family of model problems we term in context language learning (ICLL) We evaluate a diverse set of neural sequence models on regular ICLL tasks.
  arXiv  Detail & Related papers  (2024-01-23T18:59:21Z)
• The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks [64.08042492426992]
  We introduce the Expressive Memory (ELM) neuron model, a biologically inspired model of a cortical neuron. Our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. We evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets.
  arXiv  Detail & Related papers  (2023-06-14T13:34:13Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Bayesian Neural Network Language Modeling for Speech Recognition [59.681758762712754]
  State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex. In this paper, an overarching full Bayesian learning framework is proposed to account for the underlying uncertainty in LSTM-RNN and Transformer LMs.
  arXiv  Detail & Related papers  (2022-08-28T17:50:19Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.