SoftWater: Class-Aware Rate Allocation for Softmax Quantization
- URL: http://arxiv.org/abs/2608.12026v1
- Date: Wed, 12 Aug 2026 13:06:50 GMT
- Title: SoftWater: Class-Aware Rate Allocation for Softmax Quantization
- Abstract summary: Post-training quantization pipelines routinely leave the softmax output layer in high precision.<n>In small LLMs with modern vocabularies, the head holds 15--30% of all parameters, so a nominal 2-bit'' model with an fp16 head can store several times as many bits per weight.<n>We pose softmax-layer quantization as a rate-distortion problem under the KL divergence between the original and quantized output distributions.
- Score: 5.213778368155993
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Post-training quantization pipelines routinely leave the softmax output layer in high precision. Yet in small LLMs with modern vocabularies, the head holds 15--30\% of all parameters, so a nominal ``2-bit'' model with an fp16 head can store several times as many bits per weight. We pose softmax-layer quantization as a rate-distortion problem under the KL divergence between the original and quantized output distributions. A second-order analysis reveals a class-aware geometry: quantization error is weighted jointly by feature covariance and class-specific softmax curvature. A separability approximation replaces the $Kn\times Kn$ Cholesky with one $n\times n$ factorization rescaled per class, making the lattice encodable by successive interference cancellation, with both statistics from a single forward pass. The resulting method, SoftWater, gives fine grids to frequent, low-variance classes and coarse grids to rare ones, a large gap under Zipfian token distributions. Across five models from 1B to 32B, SoftWater outperforms the released WaterSIC quantizer (near-optimal under linear-layer WMSE but not output KL) at matched head rates on 59 of 60 test points, using none of that pipeline's refinements and cutting head-induced KL by $6.5\times$--$8.3\times$ at 2 bits. On Llama-3.2-1B-Instruct with quantized bodies, a 2-bit head removes 45--60\% of stored bytes for a $2.9$--$3.7\%$ perplexity increase. Because the class-side statistic comes from calibration data, matching calibration to the deployment domain gives the lowest KL on that domain throughout. On a tied model, a 4-bit head is near-lossless and a 2-bit head costs under 4\% perplexity, making head quantization of such models practical.
Related papers
- Quantum Tunneling-Aware Machine Learning: Physics-Derived Noise Models for Robust Deployment [5.373588450790602]
We introduce quantum tunneling-aware machine learning (QTAML)<n>We derive the deployment-time weight-error distribution from first principles using the Wentzel-Kramers-Brillouin (WKB) approximation.<n>We package these structural properties into a single deployment-time algorithm, Tunneling-Aware Compensation (TAC), that combines closed-form mean correction with an optimal layer-budget allocation.
arXiv Detail & Related papers (2026-05-30T14:21:16Z) - QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scaling [0.0]
Scalar post-training quantizers discard pairwise coordinate structure within weight rows.<n>We introduce QAM-W (Quadrature Amplitude Modulation for Weights), a variant that recovers this structure.<n>A 3.5bpw variant is competitive on quantization-tolerant architectures.
arXiv Detail & Related papers (2026-05-25T21:28:46Z) - LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss [8.481133435038839]
Large reasoning models reach competition-level math and coding accuracy via long autoregressive decoding.<n>Weight quantization is the standard tool for acceleration, but representative recipes lose accuracy on long-decoding reasoning benchmarks.<n>We propose LookAhead Quantization (LAQuant), a layer-wise weight-only QAT method that addresses both factors without online-transform overhead.
arXiv Detail & Related papers (2026-05-09T07:35:38Z) - A Closed-Form Persistence-Landmark Pipeline for Certified Point-Cloud and Graph Classification [0.0]
PLACE (Persistence-Landmark Analytic Classification Engine) is a closed-form pipeline for classifying point clouds and graphs.<n>Three quantitative guarantees -- a margin-based excess-risk rate, a closed-form descriptor-selection rule, and a per-prediction certificate -- are derived from training labels alone.
arXiv Detail & Related papers (2026-05-04T17:15:01Z) - BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
arXiv Detail & Related papers (2026-02-04T02:54:37Z) - SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights [8.95245917088986]
Post-training quantization has emerged as the most widely used strategy for deploying large language models at low precision.<n>Current methods show perplexity degradation at bit-widths less than or equal to 4, partly because representing outliers causes precision issues.<n>We introduce SINQ to augment existing post-training quantizers with an additional second-axis scale factor and a fast Sinkhorn-Knopp-style algorithm.
arXiv Detail & Related papers (2025-09-26T21:22:54Z) - KurTail : Kurtosis-based LLM Quantization [51.24081396305435]
KurTail is a new post-training quantization scheme that mitigates outliers in the activations of large language models.<n>It offers a 13.3% boost in MMLU accuracy and a 15.5% drop in Wiki perplexity compared to QuaRot.<n>It also outperforms SpinQuant with a 2.6% MMLU gain and reduces perplexity by 2.9%, all while reducing the training cost.
arXiv Detail & Related papers (2025-03-03T12:43:06Z) - ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization [73.60493264901359]
We present a unified framework for rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings.<n>We show that ternary, 2-bit, and 3-bit quantization maintains comparable performance in the size-accuracy trade-off.<n>Considering hardware constraints, 2-bit quantization offers promising potential for memory reduction and speedup.
arXiv Detail & Related papers (2025-02-04T18:59:26Z) - SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models [61.474101404805545]
Diffusion models can generate high-quality images, but as they scale, rising memory demands and higher latency pose deployment challenges.<n>We propose SVDQuant, a new 4-bit quantization paradigm to overcome this limitation.<n>We reduce the memory usage for the 12B FLUX.1 models by 3.5$times$, achieving 3.0$times$ speedup over the 4-bit weight-only quantization (W4A16) baseline.
arXiv Detail & Related papers (2024-11-07T18:59:58Z) - FlatQuant: Flatness Matters for LLM Quantization [58.28221892035609]
We propose FlatQuant, a new post-training quantization approach that enhances the flatness of weights and activations.<n>Our approach identifies optimal affine transformations for each linear layer, calibrated in hours via a lightweight objective.<n>It achieves less than 1% accuracy drop for W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by 7.5%.
arXiv Detail & Related papers (2024-10-12T08:10:28Z) - Towards large-scale quantum optimization solvers with few qubits [59.63282173947468]
We introduce a variational quantum solver for optimizations over $m=mathcalO(nk)$ binary variables using only $n$ qubits, with tunable $k>1$.
We analytically prove that the specific qubit-efficient encoding brings in a super-polynomial mitigation of barren plateaus as a built-in feature.
arXiv Detail & Related papers (2024-01-17T18:59:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.