HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs

• URL: http://arxiv.org/abs/2501.02625v2
• Date: Sat, 01 Feb 2025 18:58:20 GMT
• Title: HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs
• Authors: Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh, Roberto L. Castro, Torsten Hoefler, Dan Alistarh,
• Abstract summary: We present HALO, a novel quantization-aware training approach for Transformers. Our approach ensures that all large matrix multiplications during the forward and backward passes are executed in lower precision. Applying to LLAMA-family models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks.
• Score: 45.37278584462772
• License:
• Abstract: Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven difficult. This is particularly the case when fine-tuning pre-trained models, which can have large weight and activation outlier values that make lower-precision optimization difficult. To address this, we present HALO, a novel quantization-aware training approach for Transformers that enables accurate and efficient low-precision training by combining 1) strategic placement of Hadamard rotations in both forward and backward passes, which mitigate outliers, 2) high-performance kernel support, and 3) FSDP integration for low-precision communication. Our approach ensures that all large matrix multiplications during the forward and backward passes are executed in lower precision. Applied to LLAMA-family models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks, while delivering up to 1.41x end-to-end speedup for full fine-tuning on RTX 4090 GPUs. HALO efficiently supports both standard and parameterefficient fine-tuning (PEFT). Our results demonstrate the first practical approach to fully quantized LLM fine-tuning that maintains accuracy in 8-bit precision, while delivering performance benefits. Code is available at \url{https://github.com/IST-DASLab/HALO}.

Related papers

• QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models [27.730213115659986]
  Language Models (LLMs) are often quantized to lower precision to reduce the memory cost and latency in inference. Traditional fine-tuning methods require backpropagation, which are error-prone in the low-precision settings. We propose the Quantized Zeroth-Order (QuZO) framework, specifically designed for fine-tuning LLMs through low-precision forward passes.
  arXiv  Detail & Related papers  (2025-02-17T22:20:31Z)
• The Power of Negative Zero: Datatype Customization for Quantized Large Language Models [5.503925076208333]
  Post-training quantization serves as one of the most hardware-efficient methods to mitigate the memory and computational demands of large language models (LLMs) In this paper, we extend the basic FP datatype to perform Redundant Zero Remapping (RaZeR) RaZeR remaps the negative zero FP encoding to a set of pre-defined special values to maximally utilize FP quantization encodings and to better fit numerical distributions.
  arXiv  Detail & Related papers  (2025-01-06T22:40:40Z)
• Progressive Mixed-Precision Decoding for Efficient LLM Inference [49.05448842542558]
  We introduce Progressive Mixed-Precision Decoding (PMPD) to address the memory-boundedness of decoding. PMPD achieves 1.4$-$12.2$times$ speedup in matrix-vector multiplications over fp16 models. Our approach delivers a throughput gain of 3.8$-$8.0$times$ over fp16 models and up to 1.54$times$ over uniform quantization approaches.
  arXiv  Detail & Related papers  (2024-10-17T11:46:33Z)
• Zeroth-Order Fine-Tuning of LLMs in Random Subspaces [66.27334633749734]
  As language models grow in size, memory demands for backpropagation increase. Zeroth-order (ZOZO) optimization methods offer a memory-efficient alternative. We show that SubZero enhances fine-tuning and achieves faster results compared to standard ZOZO approaches.
  arXiv  Detail & Related papers  (2024-10-11T17:01:43Z)
• Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores [3.6385567224218556]
  Large language models (LLMs) have been widely applied but face challenges in efficient inference. We introduce a novel bipolar-INT data format that facilitates parallel computing and supports symmetric quantization. We implement an arbitrary precision matrix multiplication scheme that decomposes and recovers at the bit level, enabling flexible precision.
  arXiv  Detail & Related papers  (2024-09-26T14:17:58Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning on Large-Language Models. We learn the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model. Our method operates for 2.7 hours with around 35GB memory for the 13B models on a single A100 GPU.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs [53.31402059062365]
  BiLLM is a groundbreaking 1-bit post-training quantization scheme tailored for pretrained large language models. It achieves for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families.
  arXiv  Detail & Related papers  (2024-02-06T09:26:34Z)
• Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models [88.80146574509195]
  Quantization is a promising approach for reducing memory overhead and accelerating inference. We propose a novel-aware quantization (ZSAQ) framework for the zero-shot quantization of various PLMs.
  arXiv  Detail & Related papers  (2023-10-20T07:09:56Z)
• Low-Precision Arithmetic for Fast Gaussian Processes [39.720581185327816]
  Low-precision arithmetic has had a transformative effect on the training of neural networks. We propose a multi-faceted approach involving conjugate gradients with re-orthogonalization, mixed precision, and preconditioning. Our approach significantly improves the numerical stability and practical performance of conjugate gradients in low-precision over a wide range of settings.
  arXiv  Detail & Related papers  (2022-07-14T12:20:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.