SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference

• URL: http://arxiv.org/abs/2303.09266v2
• Date: Mon, 8 May 2023 13:05:00 GMT
• Title: SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
• Authors: Boren Hu, Yun Zhu, Jiacheng Li, Siliang Tang
• Abstract summary: We propose a novel dynamic early exiting combined with layer skipping for BERT inference named SmartBERT. SmartBERT can adaptively skip some layers and adaptively choose whether to exit. We conduct experiments on eight classification datasets of the GLUE benchmark.
• Score: 18.456002674399244
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dynamic early exiting has been proven to improve the inference speed of the pre-trained language model like BERT. However, all samples must go through all consecutive layers before early exiting and more complex samples usually go through more layers, which still exists redundant computation. In this paper, we propose a novel dynamic early exiting combined with layer skipping for BERT inference named SmartBERT, which adds a skipping gate and an exiting operator into each layer of BERT. SmartBERT can adaptively skip some layers and adaptively choose whether to exit. Besides, we propose cross-layer contrastive learning and combine it into our training phases to boost the intermediate layers and classifiers which would be beneficial for early exiting. To keep the consistent usage of skipping gates between training and inference phases, we propose a hard weight mechanism during training phase. We conduct experiments on eight classification datasets of the GLUE benchmark. Experimental results show that SmartBERT achieves 2-3x computation reduction with minimal accuracy drops compared with BERT and our method outperforms previous methods in both efficiency and accuracy. Moreover, in some complex datasets like RTE and WNLI, we prove that the early exiting based on entropy hardly works, and the skipping mechanism is essential for reducing computation.

Related papers

• CEEBERT: Cross-Domain Inference in Early Exit BERT [5.402030962296633]
  CeeBERT learns optimal thresholds from domain-specific confidence observed at intermediate layers on the fly. CeeBERT can speed up the BERT/ALBERT models by $2times$ - $3.5times$ with minimal drop in accuracy.
  arXiv  Detail & Related papers  (2024-05-23T20:36:10Z)
• Boosting Low-Data Instance Segmentation by Unsupervised Pre-training with Saliency Prompt [103.58323875748427]
  This work offers a novel unsupervised pre-training solution for low-data regimes. Inspired by the recent success of the Prompting technique, we introduce a new pre-training method that boosts QEIS models. Experimental results show that our method significantly boosts several QEIS models on three datasets.
  arXiv  Detail & Related papers  (2023-02-02T15:49:03Z)
• Pretraining Without Attention [114.99187017618408]
  This work explores pretraining without attention by using recent advances in sequence routing based on state-space models (SSMs) BiGS is able to match BERT pretraining accuracy on GLUE and can be extended to long-form pretraining of 4096 tokens without approximation.
  arXiv  Detail & Related papers  (2022-12-20T18:50:08Z)
• TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference [54.791572981834435]
  Existing pre-trained language models (PLMs) are often computationally expensive in inference. We propose a dynamic token reduction approach to accelerate PLMs' inference, named TR-BERT. TR-BERT formulates the token reduction process as a multi-step token selection problem and automatically learns the selection strategy via reinforcement learning.
  arXiv  Detail & Related papers  (2021-05-25T02:28:51Z)
• Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup [13.50984315473865]
  We propose an efficient multi-stage layerwise training (MSLT) approach to reduce the training time of BERT. In the proposed training strategy, only top few layers participate in backward computation, while most layers only participate in forward computation. Experimental results show that the proposed method can achieve more than 110% training speedup without significant performance degradation.
  arXiv  Detail & Related papers  (2020-11-27T10:00:22Z)
• FROST: Faster and more Robust One-shot Semi-supervised Training [0.0]
  We present a one-shot semi-supervised learning method that trains up to an order of magnitude faster and is more robust than state-of-the-art methods. Our experiments demonstrate FROST's capability to perform well when the composition of the unlabeled data is unknown.
  arXiv  Detail & Related papers  (2020-11-18T18:56:03Z)
• TernaryBERT: Distillation-aware Ultra-low Bit BERT [53.06741585060951]
  We propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods.
  arXiv  Detail & Related papers  (2020-09-27T10:17:28Z)
• BERT Loses Patience: Fast and Robust Inference with Early Exit [91.26199404912019]
  We propose Patience-based Early Exit as a plug-and-play technique to improve the efficiency and robustness of a pretrained language model. Our approach improves inference efficiency as it allows the model to make a prediction with fewer layers.
  arXiv  Detail & Related papers  (2020-06-07T13:38:32Z)
• DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference [69.93692147242284]
  Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Experiments show that DeeBERT is able to save up to 40% inference time with minimal degradation in model quality.
  arXiv  Detail & Related papers  (2020-04-27T17:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.