Transformer Quality in Linear Time

• URL: http://arxiv.org/abs/2202.10447v1
• Date: Mon, 21 Feb 2022 18:59:38 GMT
• Title: Transformer Quality in Linear Time
• Abstract summary: We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker single-head attention with minimal quality loss. We then propose a linear approximation method complementary to this new layer, which is accelerator-friendly and highly competitive in quality.
• Score: 95.2692237947444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker single-head attention with minimal quality loss. We then propose a linear approximation method complementary to this new layer, which is accelerator-friendly and highly competitive in quality. The resulting model, named FLASH, matches the perplexity of improved Transformers over both short (512) and long (8K) context lengths, achieving training speedups of up to 4.9$\times$ on Wiki-40B and 12.1$\times$ on PG-19 for auto-regressive language modeling, and 4.8$\times$ on C4 for masked language modeling.

Related papers

• Linearizing Vision Transformer with Test-Time Training [54.52616866374482]
  Inheriting weights from pretrained Transformers provides an appealing shortcut, yet the fundamental representational gap between Softmax and linear attention prevents effective weight transfer.<n>We identify Test-Time Training (TTT) as a linear-complexity architecture whose two-layer dynamic formulation is structurally aligned with Softmax attention.<n>We validate our approach by linearizing Stable Diffusion 3.5 and introduce SD3.5-T$5$ (Transformer To Test Time Training)
  arXiv  Detail & Related papers  (2026-05-04T16:16:26Z)
• CSDformer: A Conversion Method for Fully Spike-Driven Transformer [11.852241487470797]
  Spike-based transformer is a novel architecture aiming to enhance the performance of spiking neural networks.<n>We propose CSDformer, a novel conversion method for fully spike-driven transformers.<n>CSDformer achieves high performance under ultra-low latency, while dramatically reducing both computational complexity and training overhead.
  arXiv  Detail & Related papers  (2025-09-22T07:55:03Z)
• Forgetting Transformer: Softmax Attention with a Forget Gate [4.484298224007183]
  We name this attention mechanism Forgetting Attention and the resulting model the Forgetting Transformer (FoX) FoX outperforms the Transformer on long-context language modeling, length extrapolation, and short-context downstream tasks. FoX is compatible with the FlashAttention algorithm and does not require any positional embeddings.
  arXiv  Detail & Related papers  (2025-03-03T23:35:23Z)
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models [76.64055251296548]
  We show that it is feasible to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources. The resulting hybrid model, which incorporates a quarter of the attention layers, achieves performance comparable to the original Transformer in chat benchmarks.
  arXiv  Detail & Related papers  (2024-08-27T17:56:11Z)
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length [49.88826673324244]
  This work describes a hardware-efficient algorithm for training linear transformers with the delta rule. We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines.
  arXiv  Detail & Related papers  (2024-06-10T17:24:42Z)
• DeepNet: Scaling Transformers to 1,000 Layers [106.33669415337135]
  We introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer. In-depth theoretical analysis shows that model updates can be bounded in a stable way. We successfully scale Transformers up to 1,000 layers without difficulty, which is one order of magnitude deeper than previous deep Transformers.
  arXiv  Detail & Related papers  (2022-03-01T15:36:38Z)
• Primer: Searching for Efficient Transformers for Language Modeling [79.2677566332444]
  Training and inference costs of large Transformer models have grown rapidly and become expensive. Here we aim to reduce the costs of Transformers by searching for a more efficient variant. We identify an architecture, named Primer, that has a smaller training cost than the original Transformer.
  arXiv  Detail & Related papers  (2021-09-17T17:50:39Z)
• Long-Short Transformer: Efficient Transformers for Language and Vision [97.2850205384295]
  Long-Short Transformer (Transformer-LS) is an efficient self-attention mechanism for modeling long sequences with linear complexity for both language and vision tasks. It aggregates a novel long-range attention with dynamic projection to model distant correlations and a short-term attention to capture fine-grained local correlations. Our method outperforms the state-of-the-art models on multiple tasks in language and vision domains, including the Long Range Arena benchmark, autoregressive language modeling, and ImageNet classification.
  arXiv  Detail & Related papers  (2021-07-05T18:00:14Z)
• FNet: Mixing Tokens with Fourier Transforms [0.578717214982749]
  We show that Transformer encoder architectures can be massively sped up with limited accuracy costs. We replace the self-attention sublayers with simple linear transformations that "mix" input tokens. The resulting model, which we name FNet, scales very efficiently to long inputs.
  arXiv  Detail & Related papers  (2021-05-09T03:32:48Z)
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size [41.624797099537375]
  We present a novel method for applying pretrained transformer language models. We find that our method attains better perplexity than an unmodified GPT-2 model on the PG-19 and WikiText-103 corpora.
  arXiv  Detail & Related papers  (2020-08-16T23:19:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.