Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs

• URL: http://arxiv.org/abs/2606.03026v1
• Date: Tue, 02 Jun 2026 02:03:37 GMT
• Title: Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs
• Abstract summary: Spiking language models expose activation sparsity that dense Transformer runtimes do not directly exploit.<n>We implement a C++ CPU inference runtime that treats sparse binary spike states as an execution primitive.<n>Spike-aware execution can improve CPU throughput and memory behavior for sparse language models.
• Score: 8.419155861590548
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking language models expose activation sparsity that dense Transformer runtimes do not directly exploit. This paper studies that property from a systems perspective. Building on the SymbolicLight V1 spike-gated language model family, we implement a C++ CPU inference runtime that treats sparse binary spike states as an execution primitive rather than only applying post-hoc weight compression. The runtime combines a manifest-driven weight loader, mixed row/column memory layout, AVX2/FMA kernels, per-channel symmetric INT8 quantization, and integer-domain accumulation for spike-conditioned sparse paths. On an AMD Ryzen 7 5800X, an early scalar FP32 baseline decodes at 9.5 tokens/s. Mixed-layout AVX2 FP32 raises this to 14.7 tokens/s, and AVX2 INT8 reaches 19.9 tokens/s on the same step-30k export while reducing the weight footprint from 3.49 GB to 1.06 GB. For the available 186k-step 874M-parameter INT8 export, the C++ runtime decodes at 22.63 tokens/s in a single-thread CPU benchmark, compared with 16.31 tokens/s for TinyLlama-1.1B Q8_0, 11.26 tokens/s for Falcon3-1B Q8_0, and 9.70 tokens/s for Qwen2.5-1.5B Q8_0 under llama.cpp. Thread scaling reaches 47.90 tokens/s at four CPU threads, and 512-token prefill improves from 29.86 to 94.68 tokens/s from one to eight threads. The throughput result comes with a quality cost: the SNN reports WikiText-2 perplexity 24.80, worse than the dense baselines in the same benchmark. We frame the result as an inference-systems study for sparse language runtimes, with longer-term motivation in embodied and edge agents that may benefit from local, low-core inference near sensors and actuators. Spike-aware execution can improve CPU throughput and memory behavior for sparse spiking language models, while model quality, controlled dense training baselines, embodied-task evaluation, and measured CPU energy remain open problems.

Related papers

• Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators [0.0]
  "Quantize once, deploy anywhere" is unsafe for embedded and automotive deployment.<n>We show that edge-NPU latency regimes are set by output/device-to-host transfer size rather than compute.
  arXiv  Detail & Related papers  (2026-09-14T05:29:21Z)
• VibeVoice-ASR-BitNet Technical Report [86.93881874028209]
  VibeVoice-ASR-BitNet is a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs.<n>We implement a progressive quantization-aware training strategy to preserve accuracy under aggressive compression.<n> VibeVoice-ASR-BitNet is 1.6-2.3x faster than Whisper at comparable model sizes.
  arXiv  Detail & Related papers  (2026-07-23T09:08:04Z)
• Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0 [0.08599681538174887]
  Post-training INT8 (W8A8) quantization of diffusion transformers is widely deployed as a speed optimization.<n>On consumer Ampere GPU it is frequently slower than the FP8 and NF4 alternatives it is meant to beat.<n>We close this gap with a single fused Triton INT8 GEMM.<n>INT8 thus goes from the slowest variant to the fastest.
  arXiv  Detail & Related papers  (2026-06-12T16:19:03Z)
• Litespark Inference on Consumer CPUs: Custom SIMD Kernels for Ternary Neural Networks [0.00815557531820863]
  Litespark-Inference integrates directly with Hugging-Face, achieving 9.2x faster time-to-first-token, 52x higher throughput, and 14x memory reduction.<n>Our implementation, Litespark-Inference, is pip-installable and integrates directly with Hugging-Face, achieving 9.2x faster time-to-first-token, 52x higher throughput, and 14x memory reduction.
  arXiv  Detail & Related papers  (2026-05-07T16:07:39Z)
• TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference [0.0]
  TIDE is a post-training system that attaches tiny learned at periodic checkpoint layers and routers, at inference time, selects the earliest layer whose hidden state has converged for each token.<n>On an NVIDIA A100 with DeepSeek R1 Distill 8B, TIDE achieves 100% prefill exit rate (5% of tokens exit at layer 11, the remaining layer at 31), reduces prefill latency by 7.2%, and increases single-batch throughput by 6.6%.
  arXiv  Detail & Related papers  (2026-03-22T18:58:07Z)
• GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models [0.0]
  Key-value ( KV) cache presents a significant memory bottleneck during inference in large language models.<n>We implement and evaluate INT8 quantization for KV cache compression, achieving 4$times$ memory reduction with minimal accuracy degradation.
  arXiv  Detail & Related papers  (2026-01-08T08:35:56Z)
• dParallel: Learnable Parallel Decoding for dLLMs [77.24184219948337]
  Diffusion large language models (dLLMs) offer parallel token prediction and lower inference latency.<n>Existing open-source models still require nearly token-length decoding steps to ensure performance.<n>We introduce dParallel, a simple and effective method that unlocks the inherent parallelism of dLLMs for fast sampling.
  arXiv  Detail & Related papers  (2025-09-30T16:32:52Z)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [52.079202872069835]
  Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs) have grown rapidly in size.<n>We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [23.633481089469836]
  Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.<n>We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.<n>Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
  arXiv  Detail & Related papers  (2024-05-28T22:19:30Z)
• DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables [49.965024476651706]
  DeepGEMM is a lookup table based approach for the execution of ultra low-precision convolutional neural networks on SIMD hardware. Our implementation outperforms corresponding 8-bit integer kernels by up to 1.74x on x86 platforms.
  arXiv  Detail & Related papers  (2023-04-18T15:13:10Z)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale [80.86029795281922]
  We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers. A 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation.
  arXiv  Detail & Related papers  (2022-08-15T17:08:50Z)
• Efficient Execution of Quantized Deep Learning Models: A Compiler Approach [6.616902691349208]
  A growing number of applications implement predictive functions using deep learning models. Deep learning frameworks such as TFLite, MXNet, and PyTorch enable developers to quantize models with only a small drop in accuracy. They are not well suited to execute quantized models on a variety of hardware platforms.
  arXiv  Detail & Related papers  (2020-06-18T01:38:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.