Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

• URL: http://arxiv.org/abs/2609.00363v1
• Date: Tue, 25 Aug 2026 03:05:54 GMT
• Title: Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance
• Abstract summary: Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance.<n>We find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing.<n>Four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales.
• Score: 0.10152838128195468
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance. We measure what such a suite can detect. Injecting nine faults into a reference INT8 pipeline over 8,232 layer--fault--regime cells of Qwen3-1.7B, we find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing, and by exactly one whenever it moves it at all, across 5,880 cells. A tolerance of one spacing is therefore blind to the entire class by construction: four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales. Faults that violate the accumulator's exactness preconditions, or that break operand sharing, are detected without exception, and a null fault never fires. What a tolerance-based suite of this shape establishes is therefore narrower than interchangeability: that the preconditions hold, that operands are shared, and that differences stay within one spacing. The power-of-two constraint that exposes the one detected fault is also deployable. Requantizing every weight scale to its nearest power of two makes CUTLASS and Triton agree bitwise at every linear layer (196/196 and 252/252, against 8/196 and 10/252 under the checkpoints' own scales) and yields byte-identical generated token sequences at 1.7B, 8B and 14B (8/8 prompts, against 0/8 at all three). Observed perplexity point estimates are +0.32%, -0.28% and +0.48%; the 90% intervals cover zero at the two smaller sizes but not at 14B, reaching +0.71% and +0.76%. A previously reported +157% perplexity for this intervention was an artifact of a probe that rewrote scales without requantizing the weights; separating the effects attributes 99.8% of it to the resulting weight--scale mismatch rather than to the power-of-two constraint itself.

Related papers

• Quantization Amplifies Determinism, Not Bias: Scale-Dependent Behavioral Effects of Serving-Time Weight Compression [0.0]
  quantization of mid-sized models is often considered "nearly free"<n>We examine a different question: when several answers are valid, does quantization change what a model chooses to say?<n>We serve three checkpoints (Qwen3-8B/14B/32B) at three weight precisions (W4A16 AWQ, W8A16 FP8-Marlin, and bf16)<n>We collect approximately 71,000 completions paired by prompt and seed across two custom, leak-checked prompt batteries.
  arXiv  Detail & Related papers  (2026-09-07T19:06:31Z)
• Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs [0.0]
  Inference-time steering enables behavioral control of large language models without parameter modification.<n>Post-training quantization reduces memory and compute costs for deployment.<n>We systematically study activation steering under weight-only quantization.
  arXiv  Detail & Related papers  (2026-09-06T08:41:42Z)
• Lagged Coupling: Internal Representations Become Readable Before They Become Causal [0.0]
  Internal readability systematically outruns causal efficacy, and the lag does not shrink with scale.<n>A pre-registered OLMo-2 replication preserves the direction at attenuated magnitude.
  arXiv  Detail & Related papers  (2026-09-01T10:46:42Z)
• The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference [0.10152838128195468]
  Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable.<n>We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel inside vLLM.<n>At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64).
  arXiv  Detail & Related papers  (2026-08-13T20:34:36Z)
• LegoLM: Structured Weight Sharing for Large Language Models [0.0]
  We present LegoLM, a structured weight-sharing compression framework for large language models.<n>LegoLM resolves two failure modes: distributional mismatch and outlier dominance.<n>Across GPT-2 small (124M) and Mistral-7B, LegoLM achieves +0.03% PPL degradation at 4.41X compression on Mistral-7B.
  arXiv  Detail & Related papers  (2026-08-09T11:51:14Z)
• Bimanual Manipulation Within an 8 GB Budget: Zero-Copy Sensing and Quantized ACT on an Entry-Level Jetson [35.18016233072556]
  We present a bimanual SO-101 system running entirely on an NVIDIA Jetson Orin Nano Super (8 GB)<n>We build a GStreamer capture pipeline backed by NVMM buffers that removes redundant host-device copies from three-camera sensing.<n>We train ACT and Diffusion Policy on identical demonstrations, each at its own reference budget.
  arXiv  Detail & Related papers  (2026-08-04T17:09:41Z)
• Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents [19.600181756177303]
  On $2$-bench, across two open-weight model families in dense and MoE variants, quantization indeed looks free on the standard metric.<n>Quantization amplifies the failure the model already exhibits at full precision by up to 2.5$times$ in volume.<n>A targeted error-repair prompt, run for five telecom models at every precision, removes the damage exactly and only where it lives.
  arXiv  Detail & Related papers  (2026-07-29T12:40:04Z)
• Scaffold splits hide structural-frontier failures in ADMET models [25.408600489964726]
  We introduce a label-free structural-frontier split that reserves the sparsest and most physicochemically remote scaffold groups.<n>We evaluate it on six public experimental or curated ADMET tasks.
  arXiv  Detail & Related papers  (2026-07-12T12:25:50Z)
• Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels [0.0]
  tensor- kernel correctness tests go through a fixed-shape all close-style check with hand-picked absolute and relative tolerances.<n>We ask what absolute tolerance would the kernel itself, observed under its correct implementation, justify?<n>The answer is much tighter than the current hand-picked atol.
  arXiv  Detail & Related papers  (2026-06-23T20:50:36Z)
• Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness [45.148328075418156]
  A deployed ordering agent is decomposed into a fixed taxonomy of layers.<n>The pure suite runs in CI on every change against a locked per-slice baseline.<n>We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers.
  arXiv  Detail & Related papers  (2026-06-10T05:55:13Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Fast, High-Fidelity Erasure Detection of Dual-Rail Qubits with Symmetrically Coupled Readout [28.325053893714635]
  Erasure qubits are a promising platform for implementing hardware-efficient quantum error correction.<n>We realize erasure detection with a hardware-efficient circuit consisting of a single readout resonator dispersively and symmetrically coupled to both transmons of a dual-rail qubit.<n>We achieve a median $7.2 times 10-5$ error per gate with $ 1 times 10-5$ error induced by erasure detection.
  arXiv  Detail & Related papers  (2026-04-17T17:54:23Z)
• Calibration of Drive Non-Linearity for Arbitrary-Angle Single-Qubit Gates Using Error Amplification [43.97138136532209]
  Non-linearity of qubit drive line components imposes a limit on the fidelity of single-qubit gates. We demonstrate arbitrary-angle single-qubit gates with coherence-limited errors of $2times 10-4$ and leakage below $6times 10-5$.
  arXiv  Detail & Related papers  (2022-12-02T10:34:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.