Automating Versatile Time-Series Analysis with Tiny Transformers on Embedded FPGAs

• URL: http://arxiv.org/abs/2505.17662v4
• Date: Sun, 29 Jun 2025 14:00:27 GMT
• Title: Automating Versatile Time-Series Analysis with Tiny Transformers on Embedded FPGAs
• Authors: Tianheng Ling, Chao Qian, Lukas Johannes Haßler, Gregor Schiele,
• Abstract summary: This paper presents a unified and fully automated deployment framework for Tiny Transformers on embedded FPGAs.<n>Our framework supports a compact encoder-only Transformer architecture across three representative time-series tasks.<n>Results show that our framework produces integer-only, task-specific Transformer accelerators achieving as low as 0.033 mJ per inference with millisecond latency on AMD Spartan-7.
• Score: 18.15754187896287
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based models have shown strong performance across diverse time-series tasks, but their deployment on resource-constrained devices remains challenging due to high memory and computational demand. While prior work targeting Microcontroller Units (MCUs) has explored hardware-specific optimizations, such approaches are often task-specific and limited to 8-bit fixed-point precision. Field-Programmable Gate Arrays (FPGAs) offer greater flexibility, enabling fine-grained control over data precision and architecture. However, existing FPGA-based deployments of Transformers for time-series analysis typically focus on high-density platforms with manual configuration. This paper presents a unified and fully automated deployment framework for Tiny Transformers on embedded FPGAs. Our framework supports a compact encoder-only Transformer architecture across three representative time-series tasks (forecasting, classification, and anomaly detection). It combines quantization-aware training (down to 4 bits), hardware-aware hyperparameter search using Optuna, and automatic VHDL generation for seamless deployment. We evaluate our framework on six public datasets across two embedded FPGA platforms. Results show that our framework produces integer-only, task-specific Transformer accelerators achieving as low as 0.033 mJ per inference with millisecond latency on AMD Spartan-7, while also providing insights into deployment feasibility on Lattice iCE40. All source code will be released in the GitHub repository (https://github.com/Edwina1030/TinyTransformer4TS).

Related papers

• ProTEA: Programmable Transformer Encoder Acceleration on FPGA [0.0]
  Transformer neural networks (TNNs) have been widely utilized on a diverse range of applications, including natural language processing (NLP), machine translation, and computer vision (CV) Despite the popularity of TNNs, there has been limited numbers of hardware accelerators targeting these two critical blocks. This paper introduces textitProTEA, a programmable runtime accelerator tailored for the dense computations of state-of-the-art transformer encoders.
  arXiv  Detail & Related papers  (2024-09-21T01:44:13Z)
• Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers [56.37495946212932]
  Vision transformers (ViTs) have demonstrated their superior accuracy for computer vision tasks compared to convolutional neural networks (CNNs) This work proposes Quasar-ViT, a hardware-oriented quantization-aware architecture search framework for ViTs.
  arXiv  Detail & Related papers  (2024-07-25T16:35:46Z)
• AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer [54.713778961605115]
  Vision Transformer (ViT) has become one of the most prevailing fundamental backbone networks in the computer vision community. We propose a novel non-uniform quantizer, dubbed the Adaptive Logarithm AdaLog (AdaLog) quantizer.
  arXiv  Detail & Related papers  (2024-07-17T18:38:48Z)
• Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT [19.835810073852244]
  This paper presents the design of a hardware accelerator for Transformers, optimized for on-device time-series forecasting in AIoT systems. It integrates integer-only quantization and Quantization-Aware Training with optimized hardware designs to realize 6-bit and 4-bit quantized Transformer models. Compared to an 8-bit quantized Transformer model in related studies, our 4-bit quantized Transformer model increases test loss by only 0.63%, operates up to 132.33x faster, and consumes 48.19x less energy.
  arXiv  Detail & Related papers  (2024-07-06T15:03:40Z)
• MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory [76.02294791513552]
  We propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory. Experimental results demonstrate that our MCUFormer achieves 73.62% top-1 accuracy on ImageNet for image classification with 320KB memory.
  arXiv  Detail & Related papers  (2023-10-25T18:00:26Z)
• LL-GNN: Low Latency Graph Neural Networks on FPGAs for High Energy Physics [45.666822327616046]
  This work presents a novel reconfigurable architecture for Low Graph Neural Network (LL-GNN) designs for particle detectors. The LL-GNN design advances the next generation of trigger systems by enabling sophisticated algorithms to process experimental data efficiently.
  arXiv  Detail & Related papers  (2022-09-28T12:55:35Z)
• Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization [78.18328503396057]
  Vision transformers (ViTs) are emerging with significantly improved accuracy in computer vision tasks. This work proposes an FPGA-aware automatic ViT acceleration framework based on the proposed mixed-scheme quantization.
  arXiv  Detail & Related papers  (2022-08-10T05:54:46Z)
• Bilaterally Slimmable Transformer for Elastic and Efficient Visual Question Answering [75.86788916930377]
  bilaterally slimmable Transformer (BST) integrated into arbitrary Transformer-based VQA models. One slimmed MCAN-BST submodel achieves comparable accuracy on VQA-v2. Smallest MCAN-BST submodel has 9M parameters and 0.16G FLOPs during inference.
  arXiv  Detail & Related papers  (2022-03-24T02:26:04Z)
• VAQF: Fully Automatic Software-hardware Co-design Framework for Low-bit Vision Transformer [121.85581713299918]
  We propose VAQF, a framework that builds inference accelerators on FPGA platforms for quantized Vision Transformers (ViTs) Given the model structure and the desired frame rate, VAQF will automatically output the required quantization precision for activations. This is the first time quantization has been incorporated into ViT acceleration on FPGAs.
  arXiv  Detail & Related papers  (2022-01-17T20:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.