Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon

• URL: http://arxiv.org/abs/2604.16957v1
• Date: Sat, 18 Apr 2026 10:39:28 GMT
• Title: Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
• Abstract summary: We present Open-TQ-Metal, the first implementation of fused compressed-domain attention on Apple Silicon.<n>It enables 128K-context inference for Llama 3.1 70B on a single 64GB consumer Mac.<n>Open-TQ-Metal quantizes the KV cache to int4 on the fly and computes attention directly on the compressed representation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Open-TQ-Metal, the first implementation of fused compressed-domain attention on Apple Silicon, enabling 128K-context inference for Llama 3.1 70B on a single 64GB consumer Mac -- a configuration impossible with all existing inference frameworks. Open-TQ-Metal quantizes the KV cache to int4 on the fly and computes attention directly on the compressed representation via custom Metal compute shaders, eliminating all intermediate dequantization matrices. Across 330 experiments spanning two model families (Gemma 4 31B and Llama 3.1 70B), the fused sdpa_int4 kernel achieves 48x attention speedup at 128K context over the dequantize-then-attend baseline, reduces KV cache memory from 40 GB to 12.5 GB (3.2x compression), and maintains identical top-1 token predictions to FP16 inference. We further provide the first cross-architecture analysis of KV cache quantization methods, revealing that the attention scale factor -- not model size -- determines whether angular quantization schemes like PolarQuant succeed or fail, with Gemma 4's attn_scale=1.0 amplifying directional error 25-100x more than Llama's standard 1/sqrt(d) scaling.

Related papers

• Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM [0.0]
  Minima: NVFP4 W4A4 on all 496 linear layers, GDN included.<n>A four-part mechanism study explains why NVFP4's 16-element block scaling localizes the residual stream's extreme outliers.<n>A mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize.
  arXiv  Detail & Related papers  (2026-09-03T17:04:26Z)
• RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention [0.0]
  Large mixture-of-experts (MoE) language models with 26-120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures.<n>We present RotaryQuant, a three-axis compression system that addresses all three.
  arXiv  Detail & Related papers  (2026-08-08T11:58:56Z)
• RoPE-Aware Bit Allocation for KV-Cache Quantization [52.099459337231345]
  Block-GTQ is a bit allocator for key-cache quantization built on TurboQuant-MSE.<n>It better preserves RoPE query-key logits on a ten-model diagnostic panel.<n>It runs 1.34x faster than fp16 FlashAttention2 at 128K context.
  arXiv  Detail & Related papers  (2026-06-23T00:17:48Z)
• A Controlled Study of Memory Hierarchy Transitions in Quantum Circuit Simulation on Apple M4 Pro Unified Memory Architecture [0.0]
  State-vector quantum circuit simulation is memory-bandwidth bound.<n>We address this using the Apple M4 Pro Unified Memory Architecture.<n>We show that peak streaming bandwidth does not predict simulation speedup for non-contiguous memory access patterns.
  arXiv  Detail & Related papers  (2026-05-09T08:22:55Z)
• FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression [5.385515135626162]
  We present FASQ (Flexible Accelerated Subspace Quantization), a calibration-free framework that applies product quantization to large language models.<n>On Meta-Llama-3-8B, FASQ surpasses 4-bit GPTQ and AWQ in accuracy (67-42% model size) at 37-42% model size.
  arXiv  Detail & Related papers  (2026-04-22T20:03:36Z)
• MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression [7.208745673318648]
  Existing compression methods each act on a single axis of the four-dimensional KV tensor.<n>We show that different layers respond very differently to each compression operation.<n>We propose MoE-nD, a mixture-of-experts framework that routes each layer to its own.
  arXiv  Detail & Related papers  (2026-04-20T01:20:26Z)
• Quantization Dominates Rank Reduction for KV-Cache Compression [0.0]
  quantization consistently outperforms rank reduction by 4-364 PPL depending on model and compression level.<n>We formalize this via a result showing projection damage exceeds quantization damage by 3 x 2 (2b) per direction under the softmax Fisher metric.
  arXiv  Detail & Related papers  (2026-04-13T14:06:18Z)
• ITQ3_S: High-Fidelity 3-bit LLM Inference via Interleaved Ternary Quantization with Rotation-Domain Smoothing [0.0]
  We present ITQ3_S (Interleaved Ternary Quantization -- Specialized), a novel 3-bit weight quantization format for LLMs integrating TurboQuant (TQ)<n>Conventional 3-bit methods suffer precision loss from heavy-tailed weight distributions and inter-channel outliers.<n>ITQ3_S pre-rotates the weight space via FWHT before quantization, spreading energy across the vector and inducing a near-Gaussian distribution.
  arXiv  Detail & Related papers  (2026-03-30T00:03:22Z)
• Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study [0.0]
  Cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs.<n> benchmarking four models spanning 235B to 1 trillion parameters across three architectural families.
  arXiv  Detail & Related papers  (2026-02-27T13:21:48Z)
• Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices [0.0]
  Multi-agent LLM systems on edge devices face a memory management problem.<n>On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16.<n>We address this by persisting each agent's KV cache to disk in 4-bit quantized format.
  arXiv  Detail & Related papers  (2026-02-17T05:46:20Z)
• CommVQ: Commutative Vector Quantization for KV Cache Compression [50.37946553931796]
  We propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference.<n>We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache.<n>Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook.
  arXiv  Detail & Related papers  (2025-06-23T17:50:11Z)
• Fast Matrix Multiplications for Lookup Table-Quantized LLMs [58.11584672945781]
  FLUTE is a flexible lookup table engine for LUT-quantized LLMs.<n>At batch sizes 32 and quantization group size of 128, the FLUTE kernel can be 2-4x faster than existing GEMM kernels.
  arXiv  Detail & Related papers  (2024-07-15T17:55:42Z)
• LoCoCo: Dropping In Convolutions for Long Context Compression [77.26610232994508]
  This paper presents a novel approach, Dropping In Convolutions for Long Context Compression (LoCoCo) LoCoCo employs only a fixed-size Key-Value ( KV) cache, and can enhance efficiency in both inference and fine-tuning stages.
  arXiv  Detail & Related papers  (2024-06-08T01:35:11Z)
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving [52.31791050376249]
  Quantization can accelerate large language model (LLM) inference.<n>We introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache.<n>QServe improves the maximum achievable serving of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen-721.5B by 2.4x on A100, 3.5x on L40S.
  arXiv  Detail & Related papers  (2024-05-07T17:59:30Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.