Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers

• URL: http://arxiv.org/abs/2605.20289v1
• Date: Tue, 19 May 2026 06:59:46 GMT
• Title: Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers
• Abstract summary: We propose a plug-and-play framework that implements spike-friendly approximations for Transformer nonlinearities and integrates into existing ANN-to-SNN pipelines.<n>Our method decomposes these nonlinear computations into three recurring primitives -- division, exponentiation, and $ell$ norms -- and realizes them via population groups.<n>Experiments show that selectively replacing the targeted nonlinear operators incurs less than a $1% accuracy drop across all evaluated tasks.
• Score: 32.743518029177146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: ANN-to-SNN conversion offers a practical, training-free route to spiking large language models. However, current pipelines primarily focus on spike-driven realizations for Transformer linear-algebra operations, while providing limited support for key nonlinear operators. This gap limits compatibility with neuromorphic-style execution constraints, where such nonlinearities typically require division, exponentiation, or norm computations that are not naturally supported by standard leaky integrate-and-fire dynamics. To solve this problem, we propose a plug-and-play framework that implements spike-friendly approximations for Transformer nonlinearities and integrates into existing ANN-to-SNN pipelines. Our method decomposes these nonlinear computations into three recurring primitives -- division, exponentiation, and $\ell_2$ norms -- and realizes them via population computation using LIF neuron groups, combined with lightweight bit-shift scaling to avoid floating-point arithmetic. By composing these primitives as modular operator blocks, our framework supports common Transformer nonlinearities (e.g., Softmax, SiLU, and normalization) without any fine-tuning. Experiments on a range of LLMs Transformers show that selectively replacing the targeted nonlinear operators incurs less than a $1\%$ accuracy drop across all evaluated tasks.

Related papers

• Reducing ANN-SNN Conversion Error via Residual Membrane Potential Alignment [52.910770415431244]
  Spiking Neural Networks (SNNs) serve as core architectures for neuromorphic computing.<n>ANN-SNN conversion circumvents issues by reusing well-trained ANN weights for low-latency, energy-efficient inference.<n>Existing conversion schemes suffer from severe accuracy drops at small timesteps, large inference delays and cumulative quantization errors.
  arXiv  Detail & Related papers  (2026-08-14T04:46:03Z)
• Rethinking Neural Nonlinearity as Gating [0.7073210405344709]
  We show that standard activations are in fact instances of a single Threshold Gating primitive.<n>For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form.<n>We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks.
  arXiv  Detail & Related papers  (2026-07-03T09:38:16Z)
• Chebyshev-Augmented One-Shot Transfer Learning for PINNs on Nonlinear Differential Equations [0.794957965474334]
  Physics-Informed Neural Networks (PINNs) offer a flexible paradigm for solving differential equations by governing governing laws into the network.<n>A limitation is instance specificity: standard PINNs typically require retraining for each new forcing term, latent/initial condition, or nonlinear problems closed-form adaptation.<n>In this paper we address the class of nonlinearities amenable to one-shot PINN transfer by combining OTL with Chebyshev surrogates.
  arXiv  Detail & Related papers  (2026-05-02T22:49:37Z)
• Linear-Nonlinear Fusion Neural Operator for Partial Differential Equations [4.892307363245757]
  We find that explicitly decoupling linear and nonlinear effects within operator mappings leads to markedly improved learning efficiency.<n>This yields a novel network structure, namely the Linear-Nonlinear Fusion Neural Operator (LNF-NO)<n>LNF-NO naturally supports multiple functional inputs and is applicable to both regular grids and irregular geometries.
  arXiv  Detail & Related papers  (2026-03-25T10:10:53Z)
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization [52.77441224845925]
  Quantization to low bitwidth is a standard approach for deploying large language models.<n>A few extreme weights and activations stretch the dynamic range and reduce the effective resolution of the quantizer.<n>We derive, for the first time, closed-form optimal linear blockwise transforms for joint weight-activation quantization.
  arXiv  Detail & Related papers  (2025-11-30T16:17:34Z)
• S-Crescendo: A Nested Transformer Weaving Framework for Scalable Nonlinear System in S-Domain Representation [4.945568106952893]
  S-Crescendo is a nested transformer weaving framework that synergizes S-domain with neural operators for scalable time-domain prediction.<n>Our method achieves up to 0.99 test-set ($R2$) accuracy against HSPICE golden waveforms and simulation accelerates by up to 18(X)
  arXiv  Detail & Related papers  (2025-05-17T05:06:58Z)
• On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning [31.908611830179108]
  Transformer-based models demonstrate a remarkable ability for in-context learning (ICL), where they can adapt to unseen tasks from a few prompt examples without parameter updates.<n>Recent research has illuminated how Transformers perform ICL, showing that the optimal linear self-attention (LSA) mechanism can implement one step of gradient descent for linear least-squares objectives.
  arXiv  Detail & Related papers  (2025-01-30T07:41:20Z)
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length [49.88826673324244]
  This work describes a hardware-efficient algorithm for training linear transformers with the delta rule.<n>We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines.
  arXiv  Detail & Related papers  (2024-06-10T17:24:42Z)
• RWKV: Reinventing RNNs for the Transformer Era [54.716108899349614]
  We propose a novel model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers.
  arXiv  Detail & Related papers  (2023-05-22T13:57:41Z)
• NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference [9.329021390526124]
  Non-linear operations such as GELU, Layer normalization, and Softmax are essential yet costly building blocks of Transformer models. This paper proposes an accurate and hardware-friendly approximation framework for efficient Transformer inference.
  arXiv  Detail & Related papers  (2021-12-03T23:06:57Z)
• Learning to Solve the AC-OPF using Sensitivity-Informed Deep Neural Networks [52.32646357164739]
  We propose a deep neural network (DNN) to solve the solutions of the optimal power flow (ACOPF) The proposed SIDNN is compatible with a broad range of OPF schemes. It can be seamlessly integrated in other learning-to-OPF schemes.
  arXiv  Detail & Related papers  (2021-03-27T00:45:23Z)
• LQF: Linear Quadratic Fine-Tuning [114.3840147070712]
  We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification.
  arXiv  Detail & Related papers  (2020-12-21T06:40:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.