CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference

• URL: http://arxiv.org/abs/2508.20375v1
• Date: Thu, 28 Aug 2025 02:50:12 GMT
• Title: CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference
• Authors: Guanyu Xu, Zhiwei Hao, Li Shen, Yong Luo, Fuhui Sun, Xiaoyan Wang, Han Hu, Yonggang Wen,
• Abstract summary: CoFormer is a collaborative inference system for general transformer models.<n>CoFormer enables the efficient inference of GPT2-XL with 1.6 billion parameters on edge devices, reducing memory requirements by 76.3%.
• Score: 34.693462786320545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The impressive performance of transformer models has sparked the deployment of intelligent applications on resource-constrained edge devices. However, ensuring high-quality service for real-time edge systems is a significant challenge due to the considerable computational demands and resource requirements of these models. Existing strategies typically either offload transformer computations to other devices or directly deploy compressed models on individual edge devices. These strategies, however, result in either considerable communication overhead or suboptimal trade-offs between accuracy and efficiency. To tackle these challenges, we propose a collaborative inference system for general transformer models, termed CoFormer. The central idea behind CoFormer is to exploit the divisibility and integrability of transformer. An off-the-shelf large transformer can be decomposed into multiple smaller models for distributed inference, and their intermediate results are aggregated to generate the final output. We formulate an optimization problem to minimize both inference latency and accuracy degradation under heterogeneous hardware constraints. DeBo algorithm is proposed to first solve the optimization problem to derive the decomposition policy, and then progressively calibrate decomposed models to restore performance. We demonstrate the capability to support a wide range of transformer models on heterogeneous edge devices, achieving up to 3.1$\times$ inference speedup with large transformer models. Notably, CoFormer enables the efficient inference of GPT2-XL with 1.6 billion parameters on edge devices, reducing memory requirements by 76.3\%. CoFormer can also reduce energy consumption by approximately 40\% while maintaining satisfactory inference performance.

Related papers

• SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices [72.0937240883345]
  Recent advances in diffusion transformers (DiTs) have set new standards in image generation, yet remain impractical for on-device deployment.<n>We present an efficient DiT framework tailored for mobile and edge devices that achieves transformer-level generation quality under strict resource constraints.
  arXiv  Detail & Related papers  (2026-01-13T07:46:46Z)
• Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge [28.969380251735924]
  Large-scale transformers are central to modern semantic communication, yet their high computational and communication costs hinder deployment on resource-constrained edge devices.<n>This paper introduces a training-free framework for adaptive token merging, a novel mechanism that compresses transformer representations at runtime.<n>Our approach couples merging directly to input redundancy, enabling data-dependent adaptation that balances efficiency and task relevance without retraining.
  arXiv  Detail & Related papers  (2025-09-12T04:11:59Z)
• PRISM: Distributed Inference for Foundation Models at Edge [73.54372283220444]
  PRISM is a communication-efficient and compute-aware strategy for distributed Transformer inference on edge devices.<n>We evaluate PRISM on ViT, BERT, and GPT-2 across diverse datasets.
  arXiv  Detail & Related papers  (2025-07-16T11:25:03Z)
• Atleus: Accelerating Transformers on the Edge Enabled by 3D Heterogeneous Manycore Architectures [18.355570259898]
  We propose the design of a 3D heterogeneous architecture referred to as Atleus.<n>Atleus incorporates heterogeneous computing resources specifically optimized to accelerate transformer models.<n>We show that Atleus outperforms existing state-of-the-art by up to 56x and 64.5x in terms of performance and energy efficiency respectively.
  arXiv  Detail & Related papers  (2025-01-16T15:11:33Z)
• Binary Event-Driven Spiking Transformer [36.815359983551986]
  Transformer-based Spiking Neural Networks (SNNs) introduce a novel event-driven self-attention paradigm.<n>We propose the Binary Event-Driven Spiking Transformer, i.e. BESTformer.<n> BESTformer suffers from a severe performance drop from its full-precision counterpart due to the limited representation capability of binarization.
  arXiv  Detail & Related papers  (2025-01-10T12:00:11Z)
• PearSAN: A Machine Learning Method for Inverse Design using Pearson Correlated Surrogate Annealing [66.27103948750306]
  PearSAN is a machine learning-assisted optimization algorithm applicable to inverse design problems with large design spaces.<n>It uses a Pearson correlated surrogate model to predict the figure of merit of the true design metric.<n>It achieves a state-of-the-art maximum design efficiency of 97%, and is at least an order of magnitude faster than previous methods.
  arXiv  Detail & Related papers  (2024-12-26T17:02:19Z)
• Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deployment [3.391499691517567]
  Transformer models have revolutionized AI tasks, but their large size hinders real-world deployment on resource-constrained and latency-critical edge devices. We propose a co-design method for efficient end-to-end edge deployment of Transformers from three aspects: algorithm, hardware, and joint optimization. Experimental results show our co-design achieves up to 2.14-49.37x throughput gains and 3.72-88.53x better energy efficiency over state-of-the-art Transformer accelerators.
  arXiv  Detail & Related papers  (2024-07-16T12:36:10Z)
• Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptation [53.835365470800916]
  We show how to efficiently and effectively transfer knowledge in a vision transformer. We propose consolidator to modify the pre-trained model with the addition of a small set of tunable parameters. Our consolidator can reach up to 7.56 better accuracy than full fine-tuning with merely 0.35% parameters.
  arXiv  Detail & Related papers  (2023-04-30T23:59:02Z)
• TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference [6.0093441900032465]
  We propose a framework that simulates transformer inference and training on a design space of accelerators. We use this simulator in conjunction with the proposed co-design technique, called TransCODE, to obtain the best-performing models. The obtained transformer-accelerator pair achieves 0.3% higher accuracy than the state-of-the-art pair.
  arXiv  Detail & Related papers  (2023-03-27T02:45:18Z)
• Full Stack Optimization of Transformer Inference: a Survey [58.55475772110702]
  Transformer models achieve superior accuracy across a wide range of applications. The amount of compute and bandwidth required for inference of recent Transformer models is growing at a significant rate. There has been an increased focus on making Transformer models more efficient.
  arXiv  Detail & Related papers  (2023-02-27T18:18:13Z)
• HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression [69.36555801766762]
  We propose a hardware-aware tensor decomposition framework, dubbed HEAT, that enables efficient exploration of the exponential space of possible decompositions. We experimentally show that our hardware-aware factorized BERT variants reduce the energy-delay product by 5.7x with less than 1.1% accuracy loss.
  arXiv  Detail & Related papers  (2022-11-30T05:31:45Z)
• EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation [104.44478403427881]
  EdgeFormer is a parameter-efficient Transformer of the encoder-decoder architecture for on-device seq2seq generation. We conduct experiments on two practical on-device seq2seq tasks: Machine Translation and Grammatical Error Correction.
  arXiv  Detail & Related papers  (2022-02-16T10:10:00Z)
• Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization [3.5862583389869487]
  State-of-the-art Transformer-based models, with gigantic parameters, are difficult to be accommodated on resource constrained embedded devices. We propose an algorithm & hardware closed-loop acceleration framework to address the deployment challenge of Transformer. Our framework can achieve 37x, 1.9x, 1.7x speedup compared to CPU, GPU and FPGA, respectively.
  arXiv  Detail & Related papers  (2021-10-19T14:57:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.