APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

• URL: http://arxiv.org/abs/2606.08761v1
• Date: Sun, 07 Jun 2026 18:01:55 GMT
• Title: APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing
• Abstract summary: W4A4 quantization promises full utilization of INT4 Cores, yet group dequantization overhead on Cores has driven existing systems to mixed-precision fallbacks.<n>We present the first systematic study of how intra-SM compute balance governs this bottleneck.
• Score: 7.218138615728212
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks. We present the first systematic study of how intra-SM compute balance governs this bottleneck. Through controlled benchmarks across four GPUs from Ampere and Ada architectures, we identify the Tensor Cores to CUDA Cores throughput ratio ($ρ$) as the primary hardware indicator: the W4A4-g128 kernel yields $2.0$--$2.5\times$ speedup on RTX~3090 ($ρ=16$) yet degrades to $0.43$--$0.47\times$ on A100 ($ρ=64$) in compute-bond scenarios, establishing W4A4 viability as platform-dependent rather than universally infeasible. Guided by this finding, we build \textbf{APEX4}, which co-designs pure INT4 GEMM kernels with $ρ$-aware granularity adaptation to mitigate the CUDA Cores dequantization bottleneck. APEX4 achieves perplexity within 0.63 of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0\%--4.4\% in zero-shot accuracy. Deployed as a drop-in replacement in unmodified vLLM, it delivers up to $1.66\times$ end-to-end speedup on L40S ($ρ=8$), and $1.78\times$ on RTX~3090 ($ρ=16$), $2.09\times$ on A40 ($ρ=16$), while recovering A100 ($ρ=64$) to $1.20$--$1.40\times$ via the mixed-granularity mode.

Related papers

• Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM [0.0]
  Minima: NVFP4 W4A4 on all 496 linear layers, GDN included.<n>A four-part mechanism study explains why NVFP4's 16-element block scaling localizes the residual stream's extreme outliers.<n>A mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize.
  arXiv  Detail & Related papers  (2026-09-03T17:04:26Z)
• Hand-Written PTX Tensor-Core GEMM Kernels: A Multi-Precision Study on NVIDIA L4 [0.7265954868708899]
  Core kernels rely on a low-level PTX pipeline built from asynchronous data movement.<n>Most application code accesses Cores indirectly through the WMMA C++ API.<n>This paper asks a focused, practical question: when does replacing WMMA with hand-written PTX actually pay off?
  arXiv  Detail & Related papers  (2026-08-10T18:13:00Z)
• BiKAN: Restoring Collapsed Basis of Binary Kolmogorov--Arnold Networks [6.860988566886594]
  Binarizing a Kolmogorov--Arnold Network (KAN) changes function space available to each layer.<n>Our proposed BiKAN addresses this critical issue by augmenting each binary KAN with selected degree-2 Walsh characters.<n>At W1A1, BiKAN reaches $99.48%$, $84.38%$, and $55.81%$ on M, 401AR-10, and 72AR-100, respectively.
  arXiv  Detail & Related papers  (2026-08-02T20:49:59Z)
• ForgettingOT: Certified Speculative Batching from Sinkhorn's Projective Forgetting [0.0]
  We show that the eigenmode of the fixed-point Jacobian $J_tstar=QP=Pstar P$ controls the strict correction tail.<n>ForgettingOT turns this nonlinear Perron--Frobenius fact into a certified executor for streams of related Sinkhorn problems.
  arXiv  Detail & Related papers  (2026-07-27T17:59:36Z)
• A Controlled Study of Memory Hierarchy Transitions in Quantum Circuit Simulation on Apple M4 Pro Unified Memory Architecture [0.0]
  State-vector quantum circuit simulation is memory-bandwidth bound.<n>We address this using the Apple M4 Pro Unified Memory Architecture.<n>We show that peak streaming bandwidth does not predict simulation speedup for non-contiguous memory access patterns.
  arXiv  Detail & Related papers  (2026-05-09T08:22:55Z)
• DualTCN: A Physics-Constrained Temporal Convolutional Network for 2 Time-Domain Marine CSEM Inversion [1.2031796234206138]
  DualTCN is the first deep-learning framework for inverting time-domain marine controlled-source electromagnetic (MCSEM) transient data.<n>The framework regresses four earth-model parameters and reconstructs profiles using a differentiable soft-step decoder.<n>The framework demonstrates high robustness to noise through curriculum-based amplitude augmentation, maintaining a mean $barR2$ of 0.858 at $pm2%$ random amplitude error.
  arXiv  Detail & Related papers  (2026-05-06T14:58:17Z)
• SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity [86.71343842875878]
  NVIDIA's 2:4 Sparse Cores deliver 2x throughput but demand strict 50% pruning.<n>Milder $(2N-2):2N$ patterns preserve accuracy yet receive no hardware support.<n>We present SlideSparse, the first system to unlock Sparse Core acceleration.
  arXiv  Detail & Related papers  (2026-03-05T14:49:16Z)
• Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10$\ imes$ [91.61519033897424]
  Native 4K video generation remains a critical challenge due to the quadratic computational explosion of full-attention as resolution increases.<n>This paper proposes a novel Transformer retrofit strategy termed $textbfT3-Video$ that significantly reduces compute requirements by optimizing their forward logic.<n>Results on 4K-VBench show that $textbfT3-Video$ substantially outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-12-15T16:25:39Z)
• Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization [77.67818998672516]
  We present the first comprehensive study of MXFP4 and NVFP4 for post-training quantization.<n>We introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the classic GPTQ quantization algorithm.<n>We show that MR-GPTQ matches or outperforms state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-09-27T09:22:21Z)
• SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models [61.474101404805545]
  Diffusion models can generate high-quality images, but as they scale, rising memory demands and higher latency pose deployment challenges.<n>We propose SVDQuant, a new 4-bit quantization paradigm to overcome this limitation.<n>We reduce the memory usage for the 12B FLUX.1 models by 3.5$times$, achieving 3.0$times$ speedup over the 4-bit weight-only quantization (W4A16) baseline.
  arXiv  Detail & Related papers  (2024-11-07T18:59:58Z)
• COMET: Towards Partical W4A4KV4 LLMs Serving [37.30529940231099]
  Quantization is a compression technology to reduce the overhead of serving large language models (LLMs) on terminal devices and in cloud data centers. We propose a novel mixed-precision quantization algorithm (FMPQ) that compresses most activations into 4-bit with negligible accuracy loss. We integrate the optimized W4Ax kernel into our inference framework, COMET, and provide efficient management to support popular LLMs.
  arXiv  Detail & Related papers  (2024-10-16T02:16:53Z)
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving [52.31791050376249]
  Quantization can accelerate large language model (LLM) inference.<n>We introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache.<n>QServe improves the maximum achievable serving of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen-721.5B by 2.4x on A100, 3.5x on L40S.
  arXiv  Detail & Related papers  (2024-05-07T17:59:30Z)
• 1$\ imes$N Block Pattern for Network Sparsity [90.43191747596491]
  We propose one novel concept of $1times N$ block sparsity pattern (block pruning) to break this limitation. Our pattern obtains about 3.0% improvements over filter pruning in the top-1 accuracy of MobileNet-V2. It also obtains 56.04ms inference savings on Cortex-A7 CPU over weight pruning.
  arXiv  Detail & Related papers  (2021-05-31T05:50:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.