Pushing the Limits of BFP on Narrow Precision LLM Inference

• URL: http://arxiv.org/abs/2502.00026v2
• Date: Fri, 07 Feb 2025 12:23:59 GMT
• Title: Pushing the Limits of BFP on Narrow Precision LLM Inference
• Authors: Hui Wang, Yuan Cheng, Xiaomeng Han, Zhengpeng Zhao, Dawei Yang, Zhe Jiang,
• Abstract summary: Block Floating Point (BFP) has proven effective in accelerating linear operations. However, nonlinear operations, such as Attention, increasingly become performance bottlenecks. We introduce a hardware-software co-design framework (DB-Attn)
• Score: 18.53712175951463
• License:
• Abstract: The substantial computational and memory demands of Large Language Models (LLMs) hinder their deployment. Block Floating Point (BFP) has proven effective in accelerating linear operations, a cornerstone of LLM workloads. However, as sequence lengths grow, nonlinear operations, such as Attention, increasingly become performance bottlenecks due to their quadratic computational complexity. These nonlinear operations are predominantly executed using inefficient floating-point formats, which renders the system challenging to optimize software efficiency and hardware overhead. In this paper, we delve into the limitations and potential of applying BFP to nonlinear operations. Given our findings, we introduce a hardware-software co-design framework (DB-Attn), including: (i) DBFP, an advanced BFP version, overcomes nonlinear operation challenges with a pivot-focus strategy for diverse data and an adaptive grouping strategy for flexible exponent sharing. (ii) DH-LUT, a novel lookup table algorithm dedicated to accelerating nonlinear operations with DBFP format. (iii) An RTL-level DBFP-based engine is implemented to support DB-Attn, applicable to FPGA and ASIC. Results show that DB-Attn provides significant performance improvements with negligible accuracy loss, achieving 74% GPU speedup on Softmax of LLaMA and 10x low overhead performance improvement over SOTA designs.

Related papers

• Optimizing Large Language Model Training Using FP4 Quantization [73.55459961002371]
  Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce costs. This work introduces the first FP4 training framework for large language models (LLMs)
  arXiv  Detail & Related papers  (2025-01-28T18:04:50Z)
• Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format [5.527166214435735]
  quantized large language models (LLMs) leverage low-bit integer (INT) weights and retain floating-point (FP) activations. This shifts the energy and latency bottlenecks towards the FP activations that are associated with costly memory accesses and computations. Existing LLM accelerators focus primarily on computation optimizations, overlooking the potential of jointly optimizing FP computations and data movement.
  arXiv  Detail & Related papers  (2024-11-24T20:59:39Z)
• PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers [4.523939613157408]
  Vision Transformers (ViTs) are designed for Field-Programmable Gate Arrays (FPGAs) ViTs' non-linear functions pose significant obstacles to efficient hardware implementation due to their complex mathematical operations. PEANO-ViT offers a novel approach to streamlining the implementation of the layer normalization layer.
  arXiv  Detail & Related papers  (2024-06-21T03:54:10Z)
• SWAT: Scalable and Efficient Window Attention-based Transformers Acceleration on FPGAs [3.302913401404089]
  Sliding window-based static sparse attention mitigates the problem by limiting the attention scope of the input tokens. We propose a dataflow-aware FPGA-based accelerator design, SWAT, that efficiently leverages the sparsity to achieve scalable performance for long input.
  arXiv  Detail & Related papers  (2024-05-27T10:25:08Z)
• Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [71.87335804334616]
  Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. The training process of Large Language Models (LLMs) generally incurs the update of significant parameters. This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
  arXiv  Detail & Related papers  (2023-10-23T16:37:59Z)
• A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloading [62.34538208323411]
  We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs) MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
  arXiv  Detail & Related papers  (2023-09-02T11:01:16Z)
• Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures [68.91874045918112]
  adapter-ALBERT is an efficient model optimization for maximal data reuse across different tasks. We demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator.
  arXiv  Detail & Related papers  (2023-03-25T14:40:59Z)
• FireFly: A High-Throughput Hardware Accelerator for Spiking Neural Networks with Efficient DSP and Memory Optimization [6.966706170499345]
  Spiking neural networks (SNNs) have been widely used due to their strong biological interpretability and high energy efficiency. Most SNN hardware implementations for field-programmable gate arrays (FPGAs) cannot meet arithmetic or memory efficiency requirements. We propose an FPGA accelerator that can process spikes generated by the firing neuron on-the-fly (FireFly)
  arXiv  Detail & Related papers  (2023-01-05T04:28:07Z)
• Efficient Graph Neural Network Inference at Large Scale [54.89457550773165]
  Graph neural networks (GNNs) have demonstrated excellent performance in a wide range of applications. Existing scalable GNNs leverage linear propagation to preprocess the features and accelerate the training and inference procedure. We propose a novel adaptive propagation order approach that generates the personalized propagation order for each node based on its topological information.
  arXiv  Detail & Related papers  (2022-11-01T14:38:18Z)
• FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Support [8.596477111386083]
  This paper builds upon an algorithmic observation that we can accelerate the training by leveraging multiple BFP precisions. We develop a flexible DNN training accelerator, dubbed FlexBlock, which supports three different BFP precision modes. We evaluate the effectiveness of FlexBlock architecture using well-known DNNs on CIFAR, ImageNet and WMT14 datasets.
  arXiv  Detail & Related papers  (2022-03-13T15:05:34Z)
• JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data [86.8949732640035]
  We propose JUMBO, an MBO algorithm that sidesteps limitations by querying additional data. We show that it achieves no-regret under conditions analogous to GP-UCB. Empirically, we demonstrate significant performance improvements over existing approaches on two real-world optimization problems.
  arXiv  Detail & Related papers  (2021-06-02T05:03:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.