Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length

• URL: http://arxiv.org/abs/2111.09645v1
• Date: Thu, 18 Nov 2021 11:58:19 GMT
• Title: Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length
• Authors: Shira Guskin, Moshe Wasserblat, Ke Ding, Gyuwan Kim
• Abstract summary: TinyBERT addresses the computational efficiency by self-distilling BERT into a smaller transformer representation. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches.
• Score: 2.8770761243361593
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. TinyBERT addresses the computational efficiency by self-distilling BERT into a smaller transformer representation having fewer layers and smaller internal embedding. However, TinyBERT's performance drops when we reduce the number of layers by 50%, and drops even more abruptly when we reduce the number of layers by 75% for advanced NLP tasks such as span question answering. Additionally, a separate model must be trained for each inference scenario with its distinct computational budget. In this work we present Dynamic-TinyBERT, a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with <1% loss-drop). Upon publication, the code to reproduce our work will be open-sourced.

Related papers

• Sensi-BERT: Towards Sensitivity Driven Fine-Tuning for Parameter-Efficient BERT [6.029590006321152]
  We present Sensi-BERT, a sensitivity driven efficient fine-tuning of BERT models for downstream tasks. Our experiments show the efficacy of Sensi-BERT across different downstream tasks including MNLI, QQP, QNLI, SST-2 and SQuAD.
  arXiv  Detail & Related papers  (2023-07-14T17:24:15Z)
• oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes [82.99830498937729]
  oBERTa is an easy-to-use set of language models for Natural Language Processing. It allows NLP practitioners to obtain between 3.8 and 24.3 times faster models without expertise in model compression. We explore the use of oBERTa on seven representative NLP tasks.
  arXiv  Detail & Related papers  (2023-03-30T01:37:19Z)
• Gradient-Free Structured Pruning with Unlabeled Data [57.999191898036706]
  We propose a gradient-free structured pruning framework that uses only unlabeled data. Up to 40% of the original FLOP count can be reduced with less than a 4% accuracy loss across all tasks considered.
  arXiv  Detail & Related papers  (2023-03-07T19:12:31Z)
• QuaLA-MiniLM: a Quantized Length Adaptive MiniLM [5.36703735486629]
  Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. A knowledge distillation approach addresses the computational efficiency by self-distilling BERT into a smaller transformer representation having fewer layers and smaller internal embedding. Dynamic-TinyBERT tackles both limitations by partially implementing the Length Adaptive Transformer (LAT) technique onto TinyBERT, achieving x3 speedup over BERT-base with minimal accuracy loss. We use MiniLM distillation jointly with the LAT method, and we further enhance the efficiency by applying low-bit quantization.
  arXiv  Detail & Related papers  (2022-10-31T07:42:52Z)
• TangoBERT: Reducing Inference Cost by using Cascaded Architecture [9.496399437260678]
  We present TangoBERT, a cascaded model architecture in which instances are first processed by an efficient but less accurate first tier model. The decision of whether to apply the second tier model is based on a confidence score produced by the first tier model. We report TangoBERT inference CPU speedup on four text classification GLUE tasks and on one reading comprehension task.
  arXiv  Detail & Related papers  (2022-04-13T09:45:08Z)
• TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference [54.791572981834435]
  Existing pre-trained language models (PLMs) are often computationally expensive in inference. We propose a dynamic token reduction approach to accelerate PLMs' inference, named TR-BERT. TR-BERT formulates the token reduction process as a multi-step token selection problem and automatically learns the selection strategy via reinforcement learning.
  arXiv  Detail & Related papers  (2021-05-25T02:28:51Z)
• I-BERT: Integer-only BERT Quantization [78.43819756382103]
  We propose I-BERT, a novel quantization scheme for Transformer based models. I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline.
  arXiv  Detail & Related papers  (2021-01-05T02:42:58Z)
• TernaryBERT: Distillation-aware Ultra-low Bit BERT [53.06741585060951]
  We propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods.
  arXiv  Detail & Related papers  (2020-09-27T10:17:28Z)
• DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference [69.93692147242284]
  Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Experiments show that DeeBERT is able to save up to 40% inference time with minimal degradation in model quality.
  arXiv  Detail & Related papers  (2020-04-27T17:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.