Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization

• URL: http://arxiv.org/abs/2501.06663v1
• Date: Sat, 11 Jan 2025 23:29:51 GMT
• Title: Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization
• Authors: Jiayi Tian, Jinming Lu, Hai Li, Xiangwei Wang, Cong, Hao, Ian Young, Zheng Zhang,
• Abstract summary: This paper presents the first on-FPGA accelerator for end-to-end transformer training.<n>On the algorithm side, we present a bi-directional contraction flow for tensorized transformer training.<n>On the hardware side, we store all highly compressed model parameters and gradient information on chip.
• Score: 9.555456615472512
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer models have achieved state-of-the-art performance across a wide range of machine learning tasks. There is growing interest in training transformers on resource-constrained edge devices due to considerations such as privacy, domain adaptation, and on-device scientific machine learning. However, the significant computational and memory demands required for transformer training often exceed the capabilities of an edge device. Leveraging low-rank tensor compression, this paper presents the first on-FPGA accelerator for end-to-end transformer training. On the algorithm side, we present a bi-directional contraction flow for tensorized transformer training, significantly reducing the computational FLOPS and intra-layer memory costs compared to existing tensor operations. On the hardware side, we store all highly compressed model parameters and gradient information on chip, creating an on-chip-memory-only framework for each stage in training. This reduces off-chip communication and minimizes latency and energy costs. Additionally, we implement custom computing kernels for each training stage and employ intra-layer parallelism and pipe-lining to further enhance run-time and memory efficiency. Through experiments on transformer models within $36.7$ to $93.5$ MB using FP-32 data formats on the ATIS dataset, our tensorized FPGA accelerator could conduct single-batch end-to-end training on the AMD Alevo U50 FPGA, with a memory budget of less than $6$-MB BRAM and $22.5$-MB URAM. Compared to uncompressed training on the NVIDIA RTX 3090 GPU, our on-FPGA training achieves a memory reduction of $30\times$ to $51\times$. Our FPGA accelerator also achieves up to $3.6\times$ less energy cost per epoch compared with tensor Transformer training on an NVIDIA RTX 3090 GPU.

Related papers

• MegaFold: System-Level Optimizations for Accelerating Protein Structure Prediction Models [17.994632753972958]
  We present MegaFold, a cross-platform system to accelerate AF3 training.<n>We show that MegaFold reduces peak memory usage of AF3 training by up to 1.23$times$ and improves per-iteration training time by up-to 1.73$times$ and 1.62$times$ respectively.
  arXiv  Detail & Related papers  (2025-06-24T23:30:49Z)
• A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs [0.0]
  ADAPTOR is a runtime-adaptive accelerator for dense matrix computations in transformer encoders and decoders on FPGAs.<n>It incorporates efficient matrix tiling to distribute resources across FPGA platforms.<n>It achieves a speedup of 1.7 to 2.25$times$ compared to some state-of-the-art FPGA-based accelerators.
  arXiv  Detail & Related papers  (2024-11-27T08:53:19Z)
• FAMOUS: Flexible Accelerator for the Attention Mechanism of Transformer on UltraScale+ FPGAs [0.0]
  Transformer neural networks (TNNs) are being applied across a widening range of application domains, including natural language processing (NLP), machine translation, and computer vision (CV) This paper proposes textitFAMOUS, a flexible hardware accelerator for dense multi-head attention computation of TNNs on field-programmable gate arrays (FPGAs) It is optimized for high utilization of processing elements and on-chip memories to improve parallelism and reduce latency.
  arXiv  Detail & Related papers  (2024-09-21T05:25:46Z)
• Gated Slot Attention for Efficient Linear-Time Sequence Modeling [59.019501274074564]
  Gated Slot Attention (GSA) enhances Attention with Bounded-memory-Control (ABC) GSA incorporates a gating mechanism inspired by Gated Linear Attention (GLA)
  arXiv  Detail & Related papers  (2024-09-11T09:49:50Z)
• Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget [53.311109531586844]
  We demonstrate very low-cost training of large-scale T2I diffusion transformer models. We train a 1.16 billion parameter sparse transformer with only $1,890 economical cost and achieve a 12.7 FID in zero-shot generation. We aim to release our end-to-end training pipeline to further democratize the training of large-scale diffusion models on micro-budgets.
  arXiv  Detail & Related papers  (2024-07-22T17:23:28Z)
• HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis [0.1979158763744267]
  We develop an accelerator for transformers, namely, Llama 2, using high level synthesis (HLS) on Field Programmable Gate Arrays (FPGAs) We name our method HLSTransform, and the FPGA designs we synthesize with HLS achieve up to a 12.75x reduction and 8.25x reduction in energy used per token. With the lack of existing open-source FPGA accelerators for transformers, we open-source our code and document our steps for synthesis.
  arXiv  Detail & Related papers  (2024-04-29T21:26:06Z)
• A Cost-Efficient FPGA Implementation of Tiny Transformer Model using Neural ODE [0.8403582577557918]
  Transformer has been adopted to image recognition tasks and shown to outperform CNNs and RNNs while it suffers from high training cost and computational complexity. We propose a lightweight hybrid model which uses Neural ODE as a backbone instead of ResNet. The proposed model is deployed on a modest-sized FPGA device for edge computing.
  arXiv  Detail & Related papers  (2024-01-05T09:32:39Z)
• Gated Linear Attention Transformers with Hardware-Efficient Training [60.670102007737476]
  This work describes a hardware-efficient algorithm for linear attention that trades off memory movement against parallelizability. We then generalize this algorithm to a more expressive variant of linear attention with data-dependent gates. When used as a replacement for the standard attention layer in Transformers, the resulting gated linear attention Transformer is found to perform competitively.
  arXiv  Detail & Related papers  (2023-12-11T18:51:59Z)
• MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory [76.02294791513552]
  We propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory. Experimental results demonstrate that our MCUFormer achieves 73.62% top-1 accuracy on ImageNet for image classification with 320KB memory.
  arXiv  Detail & Related papers  (2023-10-25T18:00:26Z)
• Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts [60.1586169973792]
  M$3$ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE) MoE achieves better accuracy and over 80% reduction computation but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations.
  arXiv  Detail & Related papers  (2023-05-30T02:24:03Z)
• AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers [6.0093441900032465]
  Self-attention-based transformer models have achieved tremendous success in the domain of natural language processing. Previous works directly operate on large matrices involved in the attention operation, which limits hardware utilization. We propose a novel dynamic inference scheme, DynaTran, which prunes activations at runtime with low overhead.
  arXiv  Detail & Related papers  (2023-02-28T16:17:23Z)
• Mesa: A Memory-saving Training Framework for Transformers [58.78933015299703]
  We present Mesa, a memory-saving training framework for Transformers. Mesa uses exact activations during forward pass while storing a low-precision version of activations to reduce memory consumption during training. Experiments on ImageNet, CIFAR-100 and ADE20K demonstrate that Mesa can reduce half of the memory footprints during training.
  arXiv  Detail & Related papers  (2021-11-22T11:23:01Z)
• Primer: Searching for Efficient Transformers for Language Modeling [79.2677566332444]
  Training and inference costs of large Transformer models have grown rapidly and become expensive. Here we aim to reduce the costs of Transformers by searching for a more efficient variant. We identify an architecture, named Primer, that has a smaller training cost than the original Transformer.
  arXiv  Detail & Related papers  (2021-09-17T17:50:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.