Block-Skim: Efficient Question Answering for Transformer

• URL: http://arxiv.org/abs/2112.08560v1
• Date: Thu, 16 Dec 2021 01:45:33 GMT
• Title: Block-Skim: Efficient Question Answering for Transformer
• Authors: Yue Guan, Zhengyi Li, Jingwen Leng, Zhouhan Lin, Minyi Guo, Yuhao Zhu
• Abstract summary: We propose Block-Skim, which learns to skim unnecessary context in higher hidden layers to improve and accelerate the Transformer performance. We further prune the hidden states corresponding to the unnecessary positions early in lower layers, achieving significant inference-time speedup. Block-Skim improves QA models' accuracy on different datasets and achieves 3 times speedup on BERT-base model.
• Score: 25.429122678247452
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer models have achieved promising results on natural language processing (NLP) tasks including extractive question answering (QA). Common Transformer encoders used in NLP tasks process the hidden states of all input tokens in the context paragraph throughout all layers. However, different from other tasks such as sequence classification, answering the raised question does not necessarily need all the tokens in the context paragraph. Following this motivation, we propose Block-skim, which learns to skim unnecessary context in higher hidden layers to improve and accelerate the Transformer performance. The key idea of Block-Skim is to identify the context that must be further processed and those that could be safely discarded early on during inference. Critically, we find that such information could be sufficiently derived from the self-attention weights inside the Transformer model. We further prune the hidden states corresponding to the unnecessary positions early in lower layers, achieving significant inference-time speedup. To our surprise, we observe that models pruned in this way outperform their full-size counterparts. Block-Skim improves QA models' accuracy on different datasets and achieves 3 times speedup on BERT-base model.

Related papers

• MLP Can Be A Good Transformer Learner [73.01739251050076]
  Self-attention mechanism is the key of the Transformer but often criticized for its computation demands. This paper introduces a novel strategy that simplifies vision transformers and reduces computational load through the selective removal of non-essential attention layers.
  arXiv  Detail & Related papers  (2024-04-08T16:40:15Z)
• Position Prediction as an Effective Pretraining Strategy [20.925906203643883]
  We propose a novel, but surprisingly simple alternative to content reconstruction-- that of predicting locations from content, without providing positional information for it. Our approach brings improvements over strong supervised training baselines and is comparable to modern unsupervised/self-supervised pretraining methods.
  arXiv  Detail & Related papers  (2022-07-15T17:10:48Z)
• Transkimmer: Transformer Learns to Layer-wise Skim [17.188613474427054]
  One of the major computational inefficiency of Transformer-based models is that they spend identical amount of computation throughout all layers. We propose Transkimmer architecture, which learns to identify hidden state tokens that are not required by each layer. The skimmed tokens are then forwarded directly to the final output, thus reducing the computation of the successive layers.
  arXiv  Detail & Related papers  (2022-05-15T16:23:30Z)
• DoT: An efficient Double Transformer for NLP tasks with tables [3.0079490585515343]
  DoT is a double transformer model that decomposes the problem into two sub-tasks. We show that for a small drop of accuracy, DoT improves training and inference time by at least 50%.
  arXiv  Detail & Related papers  (2021-06-01T13:33:53Z)
• TransCrowd: Weakly-Supervised Crowd Counting with Transformer [56.84516562735186]
  We propose TransCrowd, which reformulates the weakly-supervised crowd counting problem from the perspective of sequence-to-count based on Transformer. Experiments on five benchmark datasets demonstrate that the proposed TransCrowd achieves superior performance compared with all the weakly-supervised CNN-based counting methods.
  arXiv  Detail & Related papers  (2021-04-19T08:12:50Z)
• IOT: Instance-wise Layer Reordering for Transformer Structures [173.39918590438245]
  We break the assumption of the fixed layer order in the Transformer and introduce instance-wise layer reordering into the model structure. Our method can also be applied to other architectures beyond Transformer.
  arXiv  Detail & Related papers  (2021-03-05T03:44:42Z)
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers [71.31712741938837]
  We show that sparse Transformers with only $O(n)$ connections per attention layer can approximate the same function class as the dense model with $n2$ connections. We also present experiments comparing different patterns/levels of sparsity on standard NLP tasks.
  arXiv  Detail & Related papers  (2020-06-08T18:30:12Z)
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing [112.2208052057002]
  We propose Funnel-Transformer which gradually compresses the sequence of hidden states to a shorter one. With comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide variety of sequence-level prediction tasks.
  arXiv  Detail & Related papers  (2020-06-05T05:16:23Z)
• DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering [22.178201429268103]
  Transformer-based QA models use input-wide self-attention across both the question and the input passage. We introduce DeFormer, which substitutes the full self-attention with question-wide and passage-wide self-attentions in the lower layers. We show DeFormer versions of BERT and XLNet can be used to speed up QA by over 4.3x and with simple distillation-based losses they incur only a 1% drop in accuracy.
  arXiv  Detail & Related papers  (2020-05-02T04:28:22Z)
• Non-Autoregressive Machine Translation with Disentangled Context Transformer [70.95181466892795]
  State-of-the-art neural machine translation models generate a translation from left to right and every step is conditioned on the previously generated tokens. We propose an attention-masking based model, called Disentangled Context (DisCo) transformer, that simultaneously generates all tokens given different contexts. Our model achieves competitive, if not better, performance compared to the state of the art in non-autoregressive machine translation while significantly reducing decoding time on average.
  arXiv  Detail & Related papers  (2020-01-15T05:32:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.