TCNCA: Temporal Convolution Network with Chunked Attention for Scalable
Sequence Processing
- URL: http://arxiv.org/abs/2312.05605v1
- Date: Sat, 9 Dec 2023 16:12:25 GMT
- Title: TCNCA: Temporal Convolution Network with Chunked Attention for Scalable
Sequence Processing
- Authors: Aleksandar Terzic, Michael Hersche, Geethan Karunaratne, Luca Benini,
Abu Sebastian, Abbas Rahimi
- Abstract summary: MEGA is a recent transformer-based architecture, which utilizes a linear recurrent operator whose parallel computation, based on the FFT, scales as $O(LlogL)$, with $L$ being the sequence length.
We build upon their approach by replacing the linear recurrence with a special temporal convolutional network which permits larger receptive field size with shallower networks, and reduces the computational complexity to $O(L)$.
We evaluate TCNCA on EnWik8 language modeling, long-range-arena (LRA) sequence classification, as well as a synthetic reasoning benchmark associative recall.
- Score: 52.64837396100988
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: MEGA is a recent transformer-based architecture, which utilizes a linear
recurrent operator whose parallel computation, based on the FFT, scales as
$O(LlogL)$, with $L$ being the sequence length. We build upon their approach by
replacing the linear recurrence with a special temporal convolutional network
which permits larger receptive field size with shallower networks, and reduces
the computational complexity to $O(L)$. The resulting model is called TCNCA, a
Temporal Convolutional Network with Chunked Attention. We evaluate TCNCA on
EnWik8 language modeling, long-range-arena (LRA) sequence classification, as
well as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCA
outperforms MEGA, reaching a lower loss with $1.37\times$/$1.24\times$ faster
forward/backward pass during training. The dilated convolutions used in TCNCA
are consistently and significantly faster operations than the FFT-based
parallelized recurrence in GPUs, making them a scalable candidate for handling
very large sequence lengths: they are up to $7.07\times$/$2.86\times$ faster in
the forward/backward pass for sequences up to 131k. Further on LRA, TCNCA
achieves, on average, $1.28\times$ speed-up during inference with similar
accuracy to what MEGA achieves. On associative recall, we find that even a
simplified version of TCNCA, without excessive multiplicative and additive
interactions, remains superior or competitive to MEGA on a range of sequence
lengths and vocabulary sizes.
Related papers
- Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [19.167604927651073]
Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.
We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.
Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
arXiv Detail & Related papers (2024-05-28T22:19:30Z) - HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM
Inference [68.59839755875252]
HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator.
We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
arXiv Detail & Related papers (2024-02-14T18:04:36Z) - Fast Multipole Attention: A Divide-and-Conquer Attention Mechanism for
Long Sequences [1.7403133838762448]
We present Fast Multipole Attention, a new attention mechanism that uses a divide-and-conquer strategy to reduce the time and memory complexity of attention for sequences of length $n$.
The hierarchical approach groups queries, keys, and values into $mathcalO( log n)$ levels of resolution, where groups at greater distances are larger in size and the weights to compute group quantities are learned.
We find empirically that the Fast Multipole Transformer performs much better than other efficient transformers in terms of memory size and accuracy.
arXiv Detail & Related papers (2023-10-18T13:40:41Z) - DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training [82.06732962485754]
FlashAttention effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU.
We introduce DISTFLASHATTN, a memory-efficient attention mechanism optimized for long-context LLMs training.
It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses.
arXiv Detail & Related papers (2023-10-05T03:47:57Z) - DeepPCR: Parallelizing Sequential Operations in Neural Networks [4.241834259165193]
We introduce DeepPCR, a novel algorithm which parallelizes typically sequential operations in order to speed up inference and training of neural networks.
DeepPCR is based on interpreting a sequence of $L$ steps as the solution of a specific system of equations, which we recover using the Parallel Cyclic Reduction algorithm.
To verify the theoretical lower complexity of the algorithm, and to identify regimes for speedup, we test the effectiveness of DeepPCR in parallelizing the forward and backward pass in multi-layer perceptrons.
arXiv Detail & Related papers (2023-09-28T10:15:30Z) - Retentive Network: A Successor to Transformer for Large Language Models [91.6652200825638]
We propose Retentive Network (RetNet) as a foundation architecture for large language models.
We theoretically derive the connection between recurrence and attention.
Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference.
arXiv Detail & Related papers (2023-07-17T16:40:01Z) - Scalable Quantum Error Correction for Surface Codes using FPGA [67.74017895815125]
A fault-tolerant quantum computer must decode and correct errors faster than they appear.
We report a distributed version of the Union-Find decoder that exploits parallel computing resources for further speedup.
The implementation employs a scalable architecture called Helios that organizes parallel computing resources into a hybrid tree-grid structure.
arXiv Detail & Related papers (2023-01-20T04:23:00Z) - SITHCon: A neural network robust to variations in input scaling on the
time dimension [0.0]
In machine learning, convolutional neural networks (CNNs) have been extremely influential in both computer vision and in recognizing patterns extended over time.
This paper introduces a Scale-Invariant Temporal History Convolution network (SITHCon) that uses a logarithmically-distributed temporal memory.
arXiv Detail & Related papers (2021-07-09T18:11:50Z) - L$^2$-GCN: Layer-Wise and Learned Efficient Training of Graph
Convolutional Networks [118.37805042816784]
Graph convolution networks (GCN) are increasingly popular in many applications, yet remain notoriously hard to train over large graph datasets.
We propose a novel efficient layer-wise training framework for GCN (L-GCN), that disentangles feature aggregation and feature transformation during training.
Experiments show that L-GCN is faster than state-of-the-arts by at least an order of magnitude, with a consistent of memory usage not dependent on dataset size.
arXiv Detail & Related papers (2020-03-30T16:37:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.