Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators

• URL: http://arxiv.org/abs/2609.16085v1
• Date: Mon, 14 Sep 2026 05:29:21 GMT
• Title: Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators
• Abstract summary: "Quantize once, deploy anywhere" is unsafe for embedded and automotive deployment.<n>We show that edge-NPU latency regimes are set by output/device-to-host transfer size rather than compute.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Eight-bit integer (INT8) post-training quantization is the default recipe for edge deployment, under a widely held assumption: INT8 makes inference faster at a small, predictable accuracy cost, and a model quantized once can be carried to any target. We test that assumption with a controlled measurement study across seven hardware classes -- ARM and x86 CPUs, a discrete GPU, an NVIDIA Jetson AGX Orin iGPU and its NVDLA cores, and two vendor NPUs (Qualcomm Hexagon HTP, DEEPX DX-M1) -- holding the ONNX artifact and the quantization scales fixed so the integer kernel or ISA is the only free variable. Portability fails on three axes. (1) The sign of the INT8 speedup is set by the CPU's dot-product ISA (ARM dotprod/SDOT, x86 VNNI): cores that have it speed up by up to 2.1x, cores that lack it slow down by 1.7x, for the identical model and runtime. (2) INT8 outputs are not portable, and the rule is an invariance rather than a gradient: FP32 predictions are bit-identical for every pair (1000/1000), while INT8 predictions agree 1000/1000 exactly when two targets share an integer kernel and 958-965/1000 whenever they do not -- independent of whether the boundary is CPU<->CPU or CPU<->accelerator, and invisible to top-1 accuracy, which is preserved. (3) Vendor NPUs own quantization: a bring-your-own QDQ graph fails silently on one NPU (external scales ignored, accuracy 0.75 -> 0.005 while it compiles, profiles and runs without error) and loudly on the other (the compiler refuses the graph), so only the vendor's native path yields a correct engine. We further show that edge-NPU latency regimes are set by output/device-to-host transfer size rather than compute, and locate the transition with a fixed-compute sweep. We release the scripts and 32 reports. "Quantize once, deploy anywhere" is unsafe for embedded and automotive deployment, where per-input determinism and redundancy matter.

Related papers

• The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference [0.10152838128195468]
  Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable.<n>We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel inside vLLM.<n>At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64).
  arXiv  Detail & Related papers  (2026-08-13T20:34:36Z)
• QSimAdv: A Late-Bound, Vendor-Agnostic Architecture for High-Performance Quantum-Circuit Simulation [1.435000056979246]
  We present QSimAdv, which makes late binding, rather than a common kernel.<n>We realize this design on NVIDIA GH200 and AMD MI250X/EPYC systems.
  arXiv  Detail & Related papers  (2026-08-11T23:17:29Z)
• Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0 [0.08599681538174887]
  Post-training INT8 (W8A8) quantization of diffusion transformers is widely deployed as a speed optimization.<n>On consumer Ampere GPU it is frequently slower than the FP8 and NF4 alternatives it is meant to beat.<n>We close this gap with a single fused Triton INT8 GEMM.<n>INT8 thus goes from the slowest variant to the fastest.
  arXiv  Detail & Related papers  (2026-06-12T16:19:03Z)
• Holding the FP8 Quality Ceiling at 8-Bit Weights and Activations: INT8 and GGUF Post-Training Quantization of Ideogram 4.0 for Consumer GPUs [0.08599681538174887]
  We study post-training quantization of Ideogram 4.0, a 9.3B flow-based diffusion transformer (DiT)<n>We evaluate every variant under schema-valid prompts produced by an expander built to Ideogram's published caption specification.<n>We release the INT8 W8A8 and GG Q4_K quantized weights on Hugging under a gated, non-commercial license.
  arXiv  Detail & Related papers  (2026-06-10T16:19:49Z)
• Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs [8.419155861590548]
  Spiking language models expose activation sparsity that dense Transformer runtimes do not directly exploit.<n>We implement a C++ CPU inference runtime that treats sparse binary spike states as an execution primitive.<n>Spike-aware execution can improve CPU throughput and memory behavior for sparse language models.
  arXiv  Detail & Related papers  (2026-06-02T02:03:37Z)
• MUXQ: Mixed-to-Uniform Precision MatriX Quantization via Low-Rank Outlier Decomposition [0.196629787330046]
  Large language models (LLMs) have achieved outstanding performance across a wide range of natural language processing tasks.<n>Existing methods, including ZeroQuant, LLM.int8() and SmoothQuant, do not fully address input-activation outliers and the associated hardware inefficiencies.<n>We propose MUXQ (Mixed-to-Uniform Quantization), which detects outlier channels in input activations and introduces a small auxiliary matrix.
  arXiv  Detail & Related papers  (2026-04-06T14:13:47Z)
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats [51.72056104795248]
  Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats.<n>This paper systematically investigates the trade-offs between FP and integer (INT) formats.<n>We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced.
  arXiv  Detail & Related papers  (2025-10-29T15:11:53Z)
• Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields [51.95157731126864]
  Machine-learning force fields can deliver accurate molecular dynamics (MD) at high computational cost.<n>This thesis aims to make MACE cheaper and faster by identifying computational bottlenecks and evaluating low-precision execution policies.
  arXiv  Detail & Related papers  (2025-10-23T14:02:34Z)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference [31.2331188304598]
  Changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses.<n>We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision.<n>Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32.
  arXiv  Detail & Related papers  (2025-06-11T08:23:53Z)
• "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization [67.3213104337679]
  Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across different formats remain unclear.<n>We conduct the most comprehensive empirical study to date, evaluating FP8, INT8, and INT4 quantization across academic benchmarks and real-world tasks.
  arXiv  Detail & Related papers  (2024-11-04T18:21:59Z)
• DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables [49.965024476651706]
  DeepGEMM is a lookup table based approach for the execution of ultra low-precision convolutional neural networks on SIMD hardware. Our implementation outperforms corresponding 8-bit integer kernels by up to 1.74x on x86 platforms.
  arXiv  Detail & Related papers  (2023-04-18T15:13:10Z)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale [80.86029795281922]
  We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers. A 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation.
  arXiv  Detail & Related papers  (2022-08-15T17:08:50Z)
• 8-bit Optimizers via Block-wise Quantization [57.25800395197516]
  Statefuls maintain statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past values. This state can be used to accelerate optimization compared to plain gradient descent but uses memory that might otherwise be allocated to model parameters. In this paper, we develop first gradients that use 8-bit statistics while maintaining the performance levels of using 32-bit gradient states.
  arXiv  Detail & Related papers  (2021-10-06T15:43:20Z)
• HAWQV3: Dyadic Neural Network Quantization [73.11579145354801]
  Current low-precision quantization algorithms often have the hidden cost of conversion back and forth from floating point to quantized integer values. We present HAWQV3, a novel mixed-precision integer-only quantization framework.
  arXiv  Detail & Related papers  (2020-11-20T23:51:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.