When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

• URL: http://arxiv.org/abs/2605.05699v1
• Date: Thu, 07 May 2026 05:44:39 GMT
• Title: When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
• Abstract summary: KV-cache quantization is framed as a quality--latency trade-off.<n>We show it is emphinverted on Apple Silicon's unified memory.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: KV-cache quantization is framed as a quality--latency trade-off. We show it is \emph{inverted} on Apple Silicon's unified memory: a single fused Metal kernel (sign-randomized FFT $+$ per-channel $λ$ $+$ per-group abs-max $+$ int4 nibble pack), exposed as a HuggingFace \texttt{Cache} subclass, runs \emph{faster than fp16} across $256$--$4096$-token prefixes on Gemma-3 1B ($-3$ to $-8\%$ ms/tok) and at short context on Qwen2.5-1.5B ($-0.7$ to $-2.6\%$ through $1$K), with $3\times$ persistent memory compression and quality preserved ($\dPPL = 0.000$ Qwen short-prompt; $+3.6$ hook $\dPPL$ Gemma). The kernel's $\sim\!25$\,ns/vec overhead is below the bandwidth savings from $3\times$ compression. The fused kernel also closes Qwen's 4-bit per-token catastrophe ($\dPPL = +7975 \to +638.6$, $12.5\times$ reduction) at $182$\,GFLOPS / $D{=}128$. Supporting findings: $\SRFT$ and $\SRHT$ are statistically indistinguishable for KV quality (we pick $\SRFT$ for mixed-radix and matrix-multiply alignment); a learned-rotation ablation surfaces a regularization role for the fixed random SRFT base (learning $R+λ$ without SRFT lowers calibration MSE $84.9\%$ vs $50.3\%$ but yields worse PPL); Householder rotations at $k{=}d/2$ reflectors are effectively lossless at $d{=}256$.

Related papers

• Asymptotically Optimal Regret for Reinforcement Learning without Horizon Dependence [54.59847568544922]
  We study horizon-free regret for finite-horizon time-homogeneous Markov decision processes with $A$ states, $A$ actions, horizon $H$, and per-trajectory total reward bounded by $1$.<n>We prove a regret upper bound [tilde O(sqrtSAK+S8A3)] with failure probability $$, where $K$ is the number of episodes and $tilde O(sqrtSAK+S3K)$ hides $mathsfpoly
  arXiv  Detail & Related papers  (2026-07-22T07:42:19Z)
• Hurwitz Quaternion Multiplicative Quantization for KV Cache Compression [19.924689669010117]
  HQMQ treats each 4-element chunk of K or V as a quaternion and quantizes its unit direction to the emphproduct $q_p cdot q_s$.<n> HQMQ delivers up to $5.05times$ KV compression, shrinking a Llama-3-70B 128k-context cache from 43 GB to 8.5 GB.
  arXiv  Detail & Related papers  (2026-05-26T20:09:17Z)
• Pruning Deep Neural Networks via the Marchenko--Pastur Distribution [0.0]
  We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets.<n>The main practical contribution is accuracy retention under short calibration and fine-tuning schedules.
  arXiv  Detail & Related papers  (2026-05-23T19:44:02Z)
• AQKA: Active Quantum Kernel Acquisition Under a Shot Budget [30.903467721896487]
  Estimating an $N times N$ quantum kernel from circuit fidelities requires $(N2 S)$ measurement shots.<n>textbfFirst, a complete regime decomposition for shot-budgeted quantum kernel learning.<n>textbfSecond, a closed-form pair-level acquisition theory.
  arXiv  Detail & Related papers  (2026-05-14T10:30:27Z)
• How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers [5.705685936981751]
  Key-value (KV) cache is the dominant memory bottleneck during Transformer inference.<n>We study how aggressively it can be compressed before multi-step reasoning degrades.
  arXiv  Detail & Related papers  (2026-04-20T08:15:17Z)
• IsoQuant: Hardware-Aligned SO(4) Isoclinic Rotations for LLM KV Cache Compression [0.4496256885343706]
  We propose a blockwise rotation framework based on quaternion algebra and the isoclinic decomposition of $SO(4)$.<n>IsoQuant achieves mean kernel-level speedups of $4.5times$--$4.7times$ over RotorQuant with peak speedups above $6times$.
  arXiv  Detail & Related papers  (2026-03-30T13:37:45Z)
• Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training [0.0]
  Attention scores in transformers are bilinear forms $S_ij = x_itop M x_j / sqrtd_h$ whose maximum magnitude governs overflow risk in low-precision training.<n>We derive a emphrank-aware concentration inequality: when the interaction matrix $M = WQ WKtop$ has rank $r ll d$, tail probabilities for $max_i,j|S_ij|$ decay as $exp(-d22/(
  arXiv  Detail & Related papers  (2026-02-21T14:29:22Z)
• Information-Computation Tradeoffs for Noiseless Linear Regression with Oblivious Contamination [65.37519531362157]
  We show that any efficient Statistical Query algorithm for this task requires VSTAT complexity at least $tildeOmega(d1/2/alpha2)$.
  arXiv  Detail & Related papers  (2025-10-12T15:42:44Z)
• Sharp Gap-Dependent Variance-Aware Regret Bounds for Tabular MDPs [54.28273395444243]
  We show that the Monotonic Value Omega (MVP) algorithm achieves a variance-aware gap-dependent regret bound of $$tildeOleft(left(sum_Delta_h(s,a)>0 fracH2 log K land mathttVar_maxtextc$.
  arXiv  Detail & Related papers  (2025-06-06T20:33:57Z)
• On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm [52.95596504632859]
  This paper establishes the convergence rate $frac1Ksum_k=1KEleft[||nabla f(xk)||_1right]leq O(fracsqrtdCK1/4)$ for AdamW measured by $ell_1$ norm.<n>We extend our result to NAdamW, an AdamW variant that employs a double-momentum mechanism, and demonstrate that it maintains the same convergence rate.
  arXiv  Detail & Related papers  (2025-05-17T05:02:52Z)
• On the $O(\frac{\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1$ Norm [54.28350823319057]
  This paper considers the RMSProp and its momentum extension and establishes the convergence rate of $frac1Tsum_k=1T.<n>Our convergence rate matches the lower bound with respect to all the coefficients except the dimension $d$.<n>Our convergence rate can be considered to be analogous to the $frac1Tsum_k=1T.
  arXiv  Detail & Related papers  (2024-02-01T07:21:32Z)
• A spectral least-squares-type method for heavy-tailed corrupted regression with unknown covariance \& heterogeneous noise [2.019622939313173]
  We revisit heavy-tailed corrupted least-squares linear regression assuming to have a corrupted $n$-sized label-feature sample of at most $epsilon n$ arbitrary outliers. We propose a near-optimal computationally tractable estimator, based on the power method, assuming no knowledge on $(Sigma,Xi) nor the operator norm of $Xi$.
  arXiv  Detail & Related papers  (2022-09-06T23:37:31Z)
• Low-Rank Approximation with $1/\epsilon^{1/3}$ Matrix-Vector Products [58.05771390012827]
  We study iterative methods based on Krylov subspaces for low-rank approximation under any Schatten-$p$ norm. Our main result is an algorithm that uses only $tildeO(k/sqrtepsilon)$ matrix-vector products.
  arXiv  Detail & Related papers  (2022-02-10T16:10:41Z)
• On the Self-Penalization Phenomenon in Feature Selection [69.16452769334367]
  We describe an implicit sparsity-inducing mechanism based on over a family of kernels. As an application, we use this sparsity-inducing mechanism to build algorithms consistent for feature selection.
  arXiv  Detail & Related papers  (2021-10-12T09:36:41Z)
• Random matrices in service of ML footprint: ternary random features with no performance loss [55.30329197651178]
  We show that the eigenspectrum of $bf K$ is independent of the distribution of the i.i.d. entries of $bf w$. We propose a novel random technique, called Ternary Random Feature (TRF) The computation of the proposed random features requires no multiplication and a factor of $b$ less bits for storage compared to classical random features.
  arXiv  Detail & Related papers  (2021-10-05T09:33:49Z)
• Random Smoothing Might be Unable to Certify $\ell_\infty$ Robustness for High-Dimensional Images [23.264535488112134]
  We show a hardness result for random smoothing to achieve adversarial robustness against attacks in the $ell_p$ ball of radius $epsilon$ when $p>2$.
  arXiv  Detail & Related papers  (2020-02-10T03:26:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.