Towards Fully FP8 GEMM LLM Training at Scale

• URL: http://arxiv.org/abs/2505.20524v1
• Date: Mon, 26 May 2025 21:04:14 GMT
• Title: Towards Fully FP8 GEMM LLM Training at Scale
• Authors: Alejandro Hernández-Cano, Dhia Garbaya, Imanol Schlag, Martin Jaggi,
• Abstract summary: Existing approaches often rely on suboptimal fine-grained FP8 kernels or fall back to higher-precision matrix multiplications.<n>We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes.<n>This enables unprecedented throughput gains, particularly at scale, while matching the downstream performance of standard BF16 training.
• Score: 77.39425361120466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the significant potential of FP8 data formats for large language model (LLM) pre-training, their adoption has been limited due to challenges in maintaining stability at scale. Existing approaches often rely on suboptimal fine-grained FP8 kernels or fall back to higher-precision matrix multiplications (GEMMs) in sensitive components, such as attention projections, compromising potential throughput gains. We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes. This enables unprecedented throughput gains, particularly at scale, while matching the downstream performance of standard BF16 training. Our architecture design reduces large outlier activations, promoting stable long-term FP8 training. In addition, we identify key metrics to monitor low-precision training and predict potential future divergences.

Related papers

• Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models [25.700481606604647]
  Experimental results demonstrate that our FP4 training scheme achieves accuracy comparable to BF16 and FP8, with smaller theoretical computational cost.<n>With the advent of next-generation hardware supporting FP4, our method sets the foundation for efficient ultra-low precision training.
  arXiv  Detail & Related papers  (2025-02-17T05:33:11Z)
• An Inquiry into Datacenter TCO for LLM Inference with FP8 [7.910301381209274]
  We analyze the computational characteristics and constraints of large language models (LLMs) inference from a TCO perspective.<n>We present a generalizable framework that enables CSPs to compare and select AI accelerators according to diverse operational requirements.
  arXiv  Detail & Related papers  (2025-02-03T05:26:22Z)
• Optimizing Large Language Model Training Using FP4 Quantization [73.55459961002371]
  Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce costs.<n>This work introduces the first FP4 training framework for large language models (LLMs)
  arXiv  Detail & Related papers  (2025-01-28T18:04:50Z)
• Scaling Laws for Floating Point Quantization Training [47.174957621592775]
  This paper explores the effects of FP quantization targets, exponent bits, mantissa bits, and the calculation of the scaling factor in FP quantization training performance of LLM models.<n>We provide the optimal exponent-mantissa bit ratio for different bit numbers, which is available for future reference by hardware manufacturers.
  arXiv  Detail & Related papers  (2025-01-05T02:30:41Z)
• Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs [4.5440077473497364]
  Large Language Models (LLMs) have attracted significant attention due to their human-like language understanding and generation capabilities. These models, characterized by their massive scale and extensive training data, continue to push the boundaries of what is possible in natural language processing. The immense computational demands associated with training such models have spurred ongoing research into optimizing the efficiency of the training process.
  arXiv  Detail & Related papers  (2024-11-10T15:19:42Z)
• "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization [67.3213104337679]
  Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across different formats remain unclear.<n>We conduct the most comprehensive empirical study to date, evaluating FP8, INT8, and INT4 quantization across academic benchmarks and real-world tasks.
  arXiv  Detail & Related papers  (2024-11-04T18:21:59Z)
• To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability [7.115739465137031]
  BrainFloat16 (BF16) precision has become the de facto standard for large language model pretraining.<n>However, prior experience with FP16, which was found to be less stable than BF16, raises concerns as to whether FP8 can be a cost-effective option for LLM training.<n>We propose new evaluation techniques and a new metric for quantifying loss landscape sharpness in autoregressive language models.
  arXiv  Detail & Related papers  (2024-05-29T02:42:23Z)
• FP8-LM: Training FP8 Large Language Models [47.17804713425323]
  In this paper, we propose a new FP8 automatic mixed-precision framework for training large language models. Experiment results show that, during the training of GPT-175B model on H100 GPU platform, our FP8 mixed-precision training framework not only achieved a remarkable 39% reduction in real memory usage but also ran 75% faster than the widely adopted BF16 framework.
  arXiv  Detail & Related papers  (2023-10-27T17:59:51Z)
• Stable and low-precision training for large-scale vision-language models [108.62077651227607]
  We introduce new methods for accelerating and stabilizing training for large language-vision models. For acceleration, we introduce SwitchBack, a linear layer for int8 quantized training which provides a speed-up of 13-25%. For stability, we analyze loss spikes and find they consistently occur 1-8 after the squared gradients become under-estimated.
  arXiv  Detail & Related papers  (2023-04-25T17:38:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.