Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention

• URL: http://arxiv.org/abs/2204.03479v1
• Date: Sun, 20 Mar 2022 20:59:13 GMT
• Title: Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention
• Abstract summary: Multi-head self-attention forms the core of Transformer networks. We propose a dynamic pruning method, which exploits the temporal stability of data across tokens to reduce inference cost. Our experiments show that we can reduce 80% of operations while maintaining the original 98.4% accuracy.
• Score: 4.925939498983408
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Multi-head self-attention forms the core of Transformer networks. However, their quadratically growing complexity with respect to the input sequence length impedes their deployment on resource-constrained edge devices. We address this challenge by proposing a dynamic pruning method, which exploits the temporal stability of data across tokens to reduce inference cost. The threshold-based method only retains significant differences between the subsequent tokens, effectively reducing the number of multiply-accumulates, as well as the internal tensor data sizes. The approach is evaluated on the Google Speech Commands Dataset for keyword spotting, and the performance is compared against the baseline Keyword Transformer. Our experiments show that we can reduce ~80% of operations while maintaining the original 98.4% accuracy. Moreover, a reduction of ~87-94% operations can be achieved when only degrading the accuracy by 1-4%, speeding up the multi-head self-attention inference by a factor of ~7.5-16.

Related papers

• Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization [0.0]
  We introduce a persistent recurrent state that observes hidden representations via cross-attention, updates itself through a GRU, and modulates subsequent processing via gated addition.<n>This module adds only 3.7% additional parameters while reducing evaluation loss from $2.438 pm 0.004$ to $1.743 pm 0.018$, corresponding to a 28.5% reduction on held-out language modeling data.
  arXiv  Detail & Related papers  (2026-09-15T14:31:43Z)
• Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers [0.0]
  We propose replacing the dense output projection in multi-head attention with a fixed, parameter-free Walsh Hadamard Transform.<n>Across different model sizes, we demonstrate that this structured substitution maintains comparable or slightly superior downstream task performance.<n>We observe that structured Hadamard-based models exhibit a steeper validation loss curve relative to training FLOPs, suggesting more favorable compute utilization during training.
  arXiv  Detail & Related papers  (2026-03-09T13:05:08Z)
• Efficient Token Compression for Vision Transformer with Spatial Information Preserved [59.79302182800274]
  Token compression is essential for reducing the computational and memory requirements of transformer models. We propose an efficient and hardware-compatible token compression method called Prune and Merge.
  arXiv  Detail & Related papers  (2025-03-30T14:23:18Z)
• Transformer Meets Twicing: Harnessing Unattended Residual Information [2.1605931466490795]
  Transformer-based deep learning models have achieved state-of-the-art performance across numerous language and vision tasks. While the self-attention mechanism has proven capable of handling complex data patterns, it has been observed that the representational capacity of the attention matrix degrades significantly across transformer layers. We propose the Twicing Attention, a novel attention mechanism that uses kernel twicing procedure in nonparametric regression to alleviate the low-pass behavior of associated NLM smoothing.
  arXiv  Detail & Related papers  (2025-03-02T01:56:35Z)
• Adaptive Pruning of Pretrained Transformer via Differential Inclusions [48.47890215458465]
  Current compression algorithms prune transformers at fixed compression ratios, requiring a unique pruning process for each ratio. We propose pruning of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter. This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure.
  arXiv  Detail & Related papers  (2025-01-06T06:34:52Z)
• Differential Transformer [99.5117269150629]
  Transformer tends to overallocate attention to irrelevant context. We introduce Diff Transformer, which amplifies attention to relevant context while canceling noise. It offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.
  arXiv  Detail & Related papers  (2024-10-07T17:57:38Z)
• LATTE: Low-Precision Approximate Attention with Head-wise Trainable Threshold for Efficient Transformer [0.0]
  We propose Low-Precision Approximate Attention with Head-wise Trainable Threshold for Efficient Transformer (LATTE) LATTE employs a headwise threshold-based filter with the low-precision dot product to reduce the computation of Multi-Head Attention (MHA) Experimental results indicate LATTE can smoothly adapt to both NLP and CV tasks, offering significant computation savings.
  arXiv  Detail & Related papers  (2024-04-11T07:23:19Z)
• Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model [10.473819332984005]
  We propose a segmented recurrent transformer (SRformer) that combines segmented (local) attention with recurrent attention. The proposed model achieves $6-22%$ higher ROUGE1 scores than a segmented transformer and outperforms other recurrent transformer approaches.
  arXiv  Detail & Related papers  (2023-05-24T03:47:22Z)
• Linearizing Transformer with Key-Value Memory Bank [54.83663647680612]
  We propose MemSizer, an approach to project the source sequence into lower dimension representation. MemSizer not only achieves the same linear time complexity but also enjoys efficient recurrent-style autoregressive generation. We demonstrate that MemSizer provides an improved tradeoff between efficiency and accuracy over the vanilla transformer.
  arXiv  Detail & Related papers  (2022-03-23T18:10:18Z)
• Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT [22.904252855587348]
  We propose a fine- and coarse-granularity hybrid self-attention that reduces the cost through progressively shortening the computational sequence length in self-attention. We show that FCA offers a significantly better trade-off between accuracy and FLOPs compared to prior methods.
  arXiv  Detail & Related papers  (2022-03-17T03:33:47Z)
• Learned Token Pruning for Transformers [39.181816379061374]
  Learned Token Pruning () method reduces redundant tokens as the data passes through the different layers of a transformer. We extensively test the performance of our approach on multiple GLUE tasks. Preliminary results show up to 1.4x and 1.9x throughput improvement on Tesla T4 and Intel Haswell.
  arXiv  Detail & Related papers  (2021-07-02T09:00:13Z)
• Finetuning Pretrained Transformers into RNNs [81.72974646901136]
  Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. A linear-complexity recurrent variant has proven well suited for autoregressive generation. This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
  arXiv  Detail & Related papers  (2021-03-24T10:50:43Z)
• DA-Transformer: Distance-aware Transformer [87.20061062572391]
  DA-Transformer is a distance-aware Transformer that can exploit the real distance. In this paper, we propose DA-Transformer, which is a distance-aware Transformer that can exploit the real distance.
  arXiv  Detail & Related papers  (2020-10-14T10:09:01Z)
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing [112.2208052057002]
  We propose Funnel-Transformer which gradually compresses the sequence of hidden states to a shorter one. With comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide variety of sequence-level prediction tasks.
  arXiv  Detail & Related papers  (2020-06-05T05:16:23Z)
• Robustness Verification for Transformers [165.25112192811764]
  We develop the first robustness verification algorithm for Transformers. The certified robustness bounds computed by our method are significantly tighter than those by naive Interval Bound propagation. These bounds also shed light on interpreting Transformers as they consistently reflect the importance of different words in sentiment analysis.
  arXiv  Detail & Related papers  (2020-02-16T17:16:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.