Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0

• URL: http://arxiv.org/abs/2606.14598v1
• Date: Fri, 12 Jun 2026 16:19:03 GMT
• Title: Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0
• Abstract summary: Post-training INT8 (W8A8) quantization of diffusion transformers is widely deployed as a speed optimization.<n>On consumer Ampere GPU it is frequently slower than the FP8 and NF4 alternatives it is meant to beat.<n>We close this gap with a single fused Triton INT8 GEMM.<n>INT8 thus goes from the slowest variant to the fastest.
• Score: 0.08599681538174887
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Post-training INT8 (W8A8) quantization of diffusion transformers is widely deployed as a speed optimization, yet on consumer Ampere GPUs it is frequently slower than the FP8 and NF4 alternatives it is meant to beat. We trace this to a software artifact: the production "INT8" forward quantizes weights and activations only to immediately dequantize them back to bf16 and run a bf16 matrix multiply, never engaging the GPU's INT8 tensor cores, so the hardware's compute advantage is left entirely unrealized. We close this gap with a single fused Triton INT8 GEMM (int8xint8->int32 on Ampere tensor cores, with per-token x per-channel dequantization and bias folded into the epilogue, autotuned per GEMM shape) dropped into the Ideogram 4.0 diffusion transformer's linear layers in place of the dequantize-to-bf16 path. In the kernel, the int8xint8->int32 accumulation is bit-exact against torch._int_mm and the dequantized output matches the reference at cosine similarity 1.0 with no NaNs, running 2.8-4.2x faster than bf16 per GEMM. End to end it delivers a ~1.1x (~9-10%) speedup at 768px, and at 1024px it generates an image in 156.5 s on a single RTX 3090, faster than the single-card NF4 (164.5 s) and FP8 (172.9 s) baselines, at no measurable quality cost on these point estimates (PickScore/CLIPScore). INT8 thus goes from the slowest variant to the fastest, and 1024px becomes single-GPU feasible. The primary speed criterion (beat FP8, by ~9.5%) is comfortably met; the NF4 margin (~4.9%, single-run n=4) is within run-to-run variance we did not quantify and is best read as consistent with meeting the stretch target. We close with an honest deployment map: the win is specific to consumer Ampere, and on A100 and B200 the same kernel loses to those cards' fast native bf16/FP8 paths.

Related papers

• Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators [0.0]
  "Quantize once, deploy anywhere" is unsafe for embedded and automotive deployment.<n>We show that edge-NPU latency regimes are set by output/device-to-host transfer size rather than compute.
  arXiv  Detail & Related papers  (2026-09-14T05:29:21Z)
• Hand-Written PTX Tensor-Core GEMM Kernels: A Multi-Precision Study on NVIDIA L4 [0.7265954868708899]
  Core kernels rely on a low-level PTX pipeline built from asynchronous data movement.<n>Most application code accesses Cores indirectly through the WMMA C++ API.<n>This paper asks a focused, practical question: when does replacing WMMA with hand-written PTX actually pay off?
  arXiv  Detail & Related papers  (2026-08-10T18:13:00Z)
• Holding the FP8 Quality Ceiling at 8-Bit Weights and Activations: INT8 and GGUF Post-Training Quantization of Ideogram 4.0 for Consumer GPUs [0.08599681538174887]
  We study post-training quantization of Ideogram 4.0, a 9.3B flow-based diffusion transformer (DiT)<n>We evaluate every variant under schema-valid prompts produced by an expander built to Ideogram's published caption specification.<n>We release the INT8 W8A8 and GG Q4_K quantized weights on Hugging under a gated, non-commercial license.
  arXiv  Detail & Related papers  (2026-06-10T16:19:49Z)
• Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs [8.419155861590548]
  Spiking language models expose activation sparsity that dense Transformer runtimes do not directly exploit.<n>We implement a C++ CPU inference runtime that treats sparse binary spike states as an execution primitive.<n>Spike-aware execution can improve CPU throughput and memory behavior for sparse language models.
  arXiv  Detail & Related papers  (2026-06-02T02:03:37Z)
• IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference [11.526305104815357]
  We present IntAttention, the first fully integer, plug-and-play attention pipeline without retraining.<n>IntAttention integrates sparsity-aware clipping, a 32-entry lookup-table approximation, and direct integer normalization.<n>Our method achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines and 2.0x faster than conventional INT8 attention pipelines on Armv8 CPUs.
  arXiv  Detail & Related papers  (2025-11-26T15:46:22Z)
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats [51.72056104795248]
  Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats.<n>This paper systematically investigates the trade-offs between FP and integer (INT) formats.<n>We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced.
  arXiv  Detail & Related papers  (2025-10-29T15:11:53Z)
• Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields [51.95157731126864]
  Machine-learning force fields can deliver accurate molecular dynamics (MD) at high computational cost.<n>This thesis aims to make MACE cheaper and faster by identifying computational bottlenecks and evaluating low-precision execution policies.
  arXiv  Detail & Related papers  (2025-10-23T14:02:34Z)
• A Speed Odyssey for Deployable Quantization of LLMs [19.12232212257625]
  We introduce a hardware-centric approach in the construction of quantization algorithms. Our method, OdysseyLLM, comes with a novel W4A8 kernel implementation called FastGEMM and a combined recipe of quantization strategies. Experiments manifest the superiority of our W4A8 method which brings the actual speed boosting up to textbf4$times$ compared to Hugging Face FP16 and textbf2.23$times$ vs. the state-of-art inference engine.
  arXiv  Detail & Related papers  (2023-11-16T04:11:19Z)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale [80.86029795281922]
  We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers. A 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation.
  arXiv  Detail & Related papers  (2022-08-15T17:08:50Z)
• 8-bit Optimizers via Block-wise Quantization [57.25800395197516]
  Statefuls maintain statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past values. This state can be used to accelerate optimization compared to plain gradient descent but uses memory that might otherwise be allocated to model parameters. In this paper, we develop first gradients that use 8-bit statistics while maintaining the performance levels of using 32-bit gradient states.
  arXiv  Detail & Related papers  (2021-10-06T15:43:20Z)
• HAWQV3: Dyadic Neural Network Quantization [73.11579145354801]
  Current low-precision quantization algorithms often have the hidden cost of conversion back and forth from floating point to quantized integer values. We present HAWQV3, a novel mixed-precision integer-only quantization framework.
  arXiv  Detail & Related papers  (2020-11-20T23:51:43Z)
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
  We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization. Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
  arXiv  Detail & Related papers  (2020-06-21T08:23:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.