RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

• URL: http://arxiv.org/abs/2608.08081v1
• Date: Sat, 08 Aug 2026 11:58:56 GMT
• Title: RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
• Abstract summary: Large mixture-of-experts (MoE) language models with 26-120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures.<n>We present RotaryQuant, a three-axis compression system that addresses all three.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand. We present RotaryQuant, a three-axis compression system that addresses all three. Mixed-precision weight quantization assigns bit-widths by architectural role: 4-bit for dense layers, 2-bit for routed experts, and 8-bit for the shared expert whose high activation kurtosis resists aggressive compression. LRU expert offloading pages non-resident experts to disk under genuine memory pressure. The novel axis is IsoQuant, a KV cache compression method that applies a Walsh--Hadamard transform followed by block-diagonal SO(4) rotations to isotropize activation distributions before 3-bit scalar quantization, requiring $O(d \log d)$ operations and 256 stored parameters per head versus $O(d^2)$ and 16{,}384 for dense rotation methods. A fused four-kernel Metal GPU pipeline performs attention directly on packed 3-bit tensors without materializing full-precision KV state---a different execution model, not just a quantization scheme. The combined system fits Gemma 4-26B-A4B and Qwen3-30B-A3B within a 16\,GB budget and Nemotron-H 120B within 32\,GB, running interactively at 9--19 tok/s with near-zero perplexity degradation ($Δ$PPL $\leq +0.0012$) and 100\% retrieval accuracy at 32K context.

Related papers

• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs [0.0]
  Key-value (KV) cache has become the dominant memory cost of transformer inference.<n>Low-rank methods factor two-dimensional slices of the cache, or per-head matrices or cross-layer feature blocks, and quantization methods lower the bit-width of every entry.<n>Our method, JoLT, applies a partial Tucker decomposition that compresses only the token and feature axes while leaving the head and layer axes intact.<n>A randomized-SVD variant, FlashJoLT, delivers a 5-13x compression-time speedup at 1024-token context and matched quality.
  arXiv  Detail & Related papers  (2026-07-14T09:23:20Z)
• RoPE-Aware Bit Allocation for KV-Cache Quantization [52.099459337231345]
  Block-GTQ is a bit allocator for key-cache quantization built on TurboQuant-MSE.<n>It better preserves RoPE query-key logits on a ten-model diagnostic panel.<n>It runs 1.34x faster than fp16 FlashAttention2 at 128K context.
  arXiv  Detail & Related papers  (2026-06-23T00:17:48Z)
• HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models [0.0]
  HyperQuant is a unified post-training quantization pipeline for the weights and the KV cache of large language and diffusion transformers.<n>It outperforms the recent HIGGS scheme at every operating point from 3 to 5 bits per scalar (bps) on weights, and beats both TurboQuant and OCTOPUS on KV quantization down to 1.7 bps.<n>It compresses the linear weights 3.9x and the KV cache 3.79x at near-lossless quality.
  arXiv  Detail & Related papers  (2026-06-22T14:30:47Z)
• XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference [0.0]
  XFP determines codebook size, outlier budget, and packing per layer automatically.<n>XFP reaches 138 tok/s single-stream decode on workstation hardware.<n>For models that do not fit in the target memory envelope, we present the H-Process.
  arXiv  Detail & Related papers  (2026-05-14T13:52:31Z)
• FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression [5.385515135626162]
  We present FASQ (Flexible Accelerated Subspace Quantization), a calibration-free framework that applies product quantization to large language models.<n>On Meta-Llama-3-8B, FASQ surpasses 4-bit GPTQ and AWQ in accuracy (67-42% model size) at 37-42% model size.
  arXiv  Detail & Related papers  (2026-04-22T20:03:36Z)
• Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon [0.0]
  We present Open-TQ-Metal, the first implementation of fused compressed-domain attention on Apple Silicon.<n>It enables 128K-context inference for Llama 3.1 70B on a single 64GB consumer Mac.<n>Open-TQ-Metal quantizes the KV cache to int4 on the fly and computes attention directly on the compressed representation.
  arXiv  Detail & Related papers  (2026-04-18T10:39:28Z)
• ITQ3_S: High-Fidelity 3-bit LLM Inference via Interleaved Ternary Quantization with Rotation-Domain Smoothing [0.0]
  We present ITQ3_S (Interleaved Ternary Quantization -- Specialized), a novel 3-bit weight quantization format for LLMs integrating TurboQuant (TQ)<n>Conventional 3-bit methods suffer precision loss from heavy-tailed weight distributions and inter-channel outliers.<n>ITQ3_S pre-rotates the weight space via FWHT before quantization, spreading energy across the vector and inducing a near-Gaussian distribution.
  arXiv  Detail & Related papers  (2026-03-30T00:03:22Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• Quantized Visual Geometry Grounded Transformer [67.15451442018258]
  This paper proposes the first Quantization framework for VGGTs, namely QuantVGGT.<n>We introduce Dual-Smoothed Fine-Grained Quantization, which integrates pre-global Hadamard rotation and post-local channel smoothing.<n>We also design Noise-Filtered Diverse Sampling, which filters outliers via deep-layer statistics.
  arXiv  Detail & Related papers  (2025-09-25T15:17:11Z)
• SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models [61.474101404805545]
  Diffusion models can generate high-quality images, but as they scale, rising memory demands and higher latency pose deployment challenges.<n>We propose SVDQuant, a new 4-bit quantization paradigm to overcome this limitation.<n>We reduce the memory usage for the 12B FLUX.1 models by 3.5$times$, achieving 3.0$times$ speedup over the 4-bit weight-only quantization (W4A16) baseline.
  arXiv  Detail & Related papers  (2024-11-07T18:59:58Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.