Gated Subspace Inference for Transformer Acceleration

• URL: http://arxiv.org/abs/2605.03109v1
• Date: Mon, 04 May 2026 19:48:40 GMT
• Title: Gated Subspace Inference for Transformer Acceleration
• Abstract summary: A method is presented for accelerating inference in transformer language models.<n>The method decomposes each activation vector into a subspace component and a residual.<n>It computes the linear-layer output on the subspace component via a cached low-rank weight image at reduced memory bandwidth.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A method is presented for accelerating inference in transformer language models by exploiting the low effective rank of the token activation manifold at each layer. The method decomposes each activation vector into a subspace component and a residual, computes the linear-layer output on the subspace component via a cached low-rank weight image at reduced memory bandwidth, and applies a per-token gate that determines whether the residual correction is computed or skipped. The gate ensures that the output distribution is preserved to within a controllable tolerance. Validation on three model families (GPT-2 124M, GPT-J 6B, OPT 6.7B) on AMD MI300X demonstrates effective speedups of 3.0x to 10.5x on linear-layer weight reads with perplexity ratios below 1.00 and top-1 token agreement above 98%. The method requires no retraining, no architectural modification, and no approximation of the attention mechanism. At the operating point (k = 256, ε = 0.05) on GPT-J 14 6B, the accelerated model produces character-for-character identical output to the baseline.

Related papers

• Accelerating Diffusion Transformers with Gaussian Process Rectified Feature Cache [17.39547918764814]
  Diffusion Transformers have become the dominant paradigm in generative AI, but their high computational costs hinder real-time applications.<n>Prediction-based feature caching is widely used to accelerate diffusion transformers; however, as the number of steps increases, the deviation between its predictions and the reference full-compute trajectory gradually grows.<n>An intuitive idea is to use an online regression model to dynamically correct this deviation, but it faces the issue of label data being unavailable during the acceleration process.<n>This paper presents a statistical observation that the residuals between the features of full computation steps using caching methods and reference full-compute trajectory locally exhibit a
  arXiv  Detail & Related papers  (2026-09-05T08:54:54Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Low-Rank Ternary Adaptation for Fine-Tuning Transformers [54.3230438745856]
  Ternary transformers offer extreme memory and compute efficiency.<n>Existing low-bit LoRA-based methods cannot directly fine-tune ternary weights.<n>We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights.
  arXiv  Detail & Related papers  (2026-08-25T12:15:56Z)
• Machine Learning-Based Reconstruction for Resistive Silicon Sensors [53.365453214937496]
  We study machine-learning-based reconstruction and compression for resistive silicon sensors.<n>We use full-waveform information from correlated pads to regularise the reconstruction and extract spatial information.<n>We first introduce recurrent neural network models based on LSTM layers, which provide a proof-of-concept implementation for full-waveform reconstruction.
  arXiv  Detail & Related papers  (2026-07-13T14:03:26Z)
• HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization [48.87937677659571]
  We introduce HARP (Hadamard-preconditioned Adaptive Rotation Processor), a learnable structured two-sided processor that replaces fixed Hadamard mixing.<n>Across 2-4 bit settings on models ranging from 1B to 70B parameters, HARP improves perplexity and zero-shot accuracy over fixed RHT.<n>HarP preserves deployment efficiency, reaching 128 tok/s versus 61 tok/s for FP16.
  arXiv  Detail & Related papers  (2026-05-28T12:24:15Z)
• Diffusion Image Generation with Explicit Modeling of Data Manifold Geometry [64.61931485161833]
  Image generative models aim to sample data points from the underlying data manifold.<n>We propose the Data Manifold-aware Image diffusioN moDel (MIND)<n>MIND integrates discrete patch tokenization into the score function of a continuous diffusion model.
  arXiv  Detail & Related papers  (2026-05-25T08:43:14Z)
• Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales [0.0]
  spectral compression, block-level function replacement, rotation-based quantization, activation geometry, and adaptive early exit are studied.<n>We identify five structural properties relevant to compression.
  arXiv  Detail & Related papers  (2026-04-22T15:31:46Z)
• Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT [0.0]
  Transformer-based language models achieve strong performance across NLP tasks, but their quadratic parameter scaling with hidden dimension makes deployment on resource-constrained hardware expensive.<n>We study Matrix Product Operator decomposition as a principled compression method for transformers.<n>MPO factorises weight matrices into chains of low-rank cores, with approximation quality controlled by the bond dimension chi.
  arXiv  Detail & Related papers  (2026-03-30T14:57:47Z)
• Latent attention on masked patches for flow reconstruction [8.69419238669827]
  We introduce the Latent Attention on Masked Patches (LAMP) model, an interpretable regression-based modified vision transformer for masked flow reconstruction.<n>We show that the LAMP accurately reconstructs the full flow field from a 90%-masked and noisy input, across signal-to-noise ratios between 10 and 30,dB.
  arXiv  Detail & Related papers  (2026-03-02T16:12:40Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• Simultaneous Optimization of Efficiency and Degradation in Tunable HTL-Free Perovskite Solar Cells with MWCNT-Integrated Back Contact Using a Machine Learning-Derived Polynomial Regressor [0.8739101659113155]
  Perovskite solar cells (PSCs) without a hole transport layer (HTL) offer a cost-effective and stable alternative to conventional architectures.<n>This study presents a machine learning (ML)-driven framework to optimize the efficiency and stability of HTL-free PSCs.
  arXiv  Detail & Related papers  (2025-05-24T13:37:48Z)
• On Disentangled Training for Nonlinear Transform in Learned Image Compression [59.66885464492666]
  Learned image compression (LIC) has demonstrated superior rate-distortion (R-D) performance compared to traditional codecs.<n>Existing LIC methods overlook the slow convergence caused by compacting energy in learning nonlinear transforms.<n>We propose a linear auxiliary transform (AuxT) to disentangle energy compaction in training nonlinear transforms.
  arXiv  Detail & Related papers  (2025-01-23T15:32:06Z)
• ESPACE: Dimensionality Reduction of Activations for Model Compression [4.98131272298465]
  We propose ESPACE, an LLM compression technique based on dimensionality reduction of activations. ESPACE enables 50% compression of GPT3, Llama2, and Nemotron4 models with small accuracy degradation. ESPACE also reduces GEMM execution time and prefill inference latency on existing hardware.
  arXiv  Detail & Related papers  (2024-10-07T18:59:22Z)
• Adaptive Multi-step Refinement Network for Robust Point Cloud Registration [82.64560249066734]
  Point Cloud Registration estimates the relative rigid transformation between two point clouds of the same scene.<n>We propose an adaptive multi-step refinement network that refines the registration quality at each step by leveraging the information from the preceding step.<n>Our method achieves state-of-the-art performance on both the 3DMatch/3DLoMatch and KITTI benchmarks.
  arXiv  Detail & Related papers  (2023-12-05T18:59:41Z)
• Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior [54.629850694790036]
  spectral-normalized identity priors (SNIP) is a structured pruning approach that penalizes an entire residual module in a Transformer model toward an identity mapping. We conduct experiments with BERT on 5 GLUE benchmark tasks to demonstrate that SNIP achieves effective pruning results while maintaining comparable performance.
  arXiv  Detail & Related papers  (2020-10-05T05:40:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.