The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference

• URL: http://arxiv.org/abs/2608.13756v2
• Date: Tue, 18 Aug 2026 06:33:51 GMT
• Title: The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference
• Abstract summary: Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable.<n>We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel inside vLLM.<n>At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64).
• Score: 0.10152838128195468
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable. We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel (CUTLASS versus Triton) inside vLLM. At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64). What makes this more than a benchmark discrepancy is an integer alibi: for shared INT8 operands under a verified no-overflow bound, the INT32 dot product is exact and order-independent, so the accumulator cannot be the source of any difference. Feeding both kernels identical operands from every linear layer of Qwen3-1.7B and 8B (196 and 252 layers), we find bit-identical outputs under power-of-two scales, confirming a pinned prediction list 196/196 and 252/252 (pre-registered at 1.7B, pinned but not blind at 8B), and observed differences of at most one bfloat16 spacing under the checkpoints' real scales. This localizes the divergence to scale application and output rounding after the exact accumulator. Applied as a probe checkpoint, the same intervention restores end-to-end bitwise agreement (8/8 and 16/16 sequences). Cross-implementation FP8 GEMM shows a different signature: both the prevalence and the magnitude of differences grow with reduction depth, while the INT8 fraction stays at parts per million and within one spacing over a 64x range of K. Teacher-forced replay ties layers to tokens: flips concentrate at small logit margins, which predict flip risk with ROC-AUC 0.94 on 16,384 positions. We will release the pre-registration, per-layer predictions, manifests with kernel-selection evidence, and a conformance procedure that turns these controls into a concrete check for kernel interchangeability.

Related papers

• Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators [0.0]
  "Quantize once, deploy anywhere" is unsafe for embedded and automotive deployment.<n>We show that edge-NPU latency regimes are set by output/device-to-host transfer size rather than compute.
  arXiv  Detail & Related papers  (2026-09-14T05:29:21Z)
• Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance [0.10152838128195468]
  Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance.<n>We find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing.<n>Four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales.
  arXiv  Detail & Related papers  (2026-08-25T03:05:54Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• qZACH-ViT: Quantization-Aware Intrinsic Explanations with Recursive Attribution-Stabilized Optimization [0.0]
  We introduce qZACH-ViT, a quantization-aware extension of the zero-token-free ZACH-ViT backbone.<n>We also introduce Recursive Attribution-Stabilized Optimization (RASO), which norm-matches classification and attribution gradients.<n>We evaluate four controlled conditions on seven MedMNIST datasets using 50 training images per class and ten fixed seeds.
  arXiv  Detail & Related papers  (2026-07-16T19:51:48Z)
• Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels [0.0]
  tensor- kernel correctness tests go through a fixed-shape all close-style check with hand-picked absolute and relative tolerances.<n>We ask what absolute tolerance would the kernel itself, observed under its correct implementation, justify?<n>The answer is much tighter than the current hand-picked atol.
  arXiv  Detail & Related papers  (2026-06-23T20:50:36Z)
• Holding the FP8 Quality Ceiling at 8-Bit Weights and Activations: INT8 and GGUF Post-Training Quantization of Ideogram 4.0 for Consumer GPUs [0.08599681538174887]
  We study post-training quantization of Ideogram 4.0, a 9.3B flow-based diffusion transformer (DiT)<n>We evaluate every variant under schema-valid prompts produced by an expander built to Ideogram's published caption specification.<n>We release the INT8 W8A8 and GG Q4_K quantized weights on Hugging under a gated, non-commercial license.
  arXiv  Detail & Related papers  (2026-06-10T16:19:49Z)
• The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference [2.954681536906518]
  KV caching is a ubiquitous optimization in autoregressive transformer inference.<n>We show that FP16 KV cache inference is fundamentally non-equivalent to recomputation.
  arXiv  Detail & Related papers  (2026-04-16T15:59:40Z)
• LUT-KAN: Segment-wise LUT Quantization for Fast KAN Inference [20.271194684947282]
  This paper introduces LUT-KAN, a segment-wise lookup-table (LUT) compilation and quantization method for PyKAN-style KAN layers.<n>LUT-KAN converts each edge function into a per-segment LUT with affine int8/uint8 quantization and linear.<n>We report accuracy, speed, and memory metrics with mean and standard deviation across multiple seeds.
  arXiv  Detail & Related papers  (2026-01-06T18:00:45Z)
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats [51.72056104795248]
  Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats.<n>This paper systematically investigates the trade-offs between FP and integer (INT) formats.<n>We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced.
  arXiv  Detail & Related papers  (2025-10-29T15:11:53Z)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference [31.2331188304598]
  Changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses.<n>We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision.<n>Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32.
  arXiv  Detail & Related papers  (2025-06-11T08:23:53Z)
• Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss [59.835032408496545]
  We propose a tile-based strategy that partitions the contrastive loss calculation into arbitrary small blocks. We also introduce a multi-level tiling strategy to leverage the hierarchical structure of distributed systems. Compared to SOTA memory-efficient solutions, it achieves a two-order-of-magnitude reduction in memory while maintaining comparable speed.
  arXiv  Detail & Related papers  (2024-10-22T17:59:30Z)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale [80.86029795281922]
  We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers. A 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation.
  arXiv  Detail & Related papers  (2022-08-15T17:08:50Z)
• I-BERT: Integer-only BERT Quantization [78.43819756382103]
  We propose I-BERT, a novel quantization scheme for Transformer based models. I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline.
  arXiv  Detail & Related papers  (2021-01-05T02:42:58Z)
• HAWQV3: Dyadic Neural Network Quantization [73.11579145354801]
  Current low-precision quantization algorithms often have the hidden cost of conversion back and forth from floating point to quantized integer values. We present HAWQV3, a novel mixed-precision integer-only quantization framework.
  arXiv  Detail & Related papers  (2020-11-20T23:51:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.