Extreme Model Compression for On-device Natural Language Understanding

• URL: http://arxiv.org/abs/2012.00124v1
• Date: Mon, 30 Nov 2020 21:47:48 GMT
• Title: Extreme Model Compression for On-device Natural Language Understanding
• Authors: Kanthashree Mysore Sathyendra, Samridhi Choudhary, Leah Nicolich-Henkin
• Abstract summary: We show our results on a large-scale, commercial NLU system trained on a varied set of intents with huge vocabulary sizes. Our approach outperforms a range of baselines and achieves a compression rate of 97.4% with less than 3.7% degradation in predictive performance.
• Score: 6.941609786551173
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we propose and experiment with techniques for extreme compression of neural natural language understanding (NLU) models, making them suitable for execution on resource-constrained devices. We propose a task-aware, end-to-end compression approach that performs word-embedding compression jointly with NLU task learning. We show our results on a large-scale, commercial NLU system trained on a varied set of intents with huge vocabulary sizes. Our approach outperforms a range of baselines and achieves a compression rate of 97.4% with less than 3.7% degradation in predictive performance. Our analysis indicates that the signal from the downstream task is important for effective compression with minimal degradation in performance.

Related papers

• LanguaShrink: Reducing Token Overhead with Psycholinguistics [8.123272461141815]
  LanguaShrink is a prompt compression framework for large language models. It reduces prompt length while preserving essential information. Compared to existing prompt compression methods, LanguaShrink improves end-to-end latency by 1.43 times.
  arXiv  Detail & Related papers  (2024-09-01T22:09:20Z)
• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression [43.048684907893104]
  This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT.
  arXiv  Detail & Related papers  (2024-03-19T17:59:56Z)
• Activations and Gradients Compression for Model-Parallel Training [85.99744701008802]
  We study how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We find that gradients require milder compression rates than activations. Experiments also show that models trained with TopK perform well only when compression is also applied during inference.
  arXiv  Detail & Related papers  (2024-01-15T15:54:54Z)
• Approximating Human-Like Few-shot Learning with GPT-based Compression [55.699707962017975]
  We seek to equip generative pre-trained models with human-like learning capabilities that enable data compression during inference. We present a novel approach that utilizes the Generative Pre-trained Transformer (GPT) to approximate Kolmogorov complexity.
  arXiv  Detail & Related papers  (2023-08-14T05:22:33Z)
• Compressing Transformer-based self-supervised models for speech processing [45.254624876127124]
  We study several commonly used compression techniques, including weight pruning, head pruning, low-rank approximation, and knowledge distillation. We report trade-off at various compression rate, including wall-clock time, the number of parameters, and the number of multiply-accumulate operations. Our results lead to a simple combination of compression techniques that improves trade-off over recent approaches.
  arXiv  Detail & Related papers  (2022-11-17T23:53:52Z)
• What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression [68.82486784654817]
  We study two popular model compression techniques including knowledge distillation and pruning. We show that compressed models are significantly less robust than their PLM counterparts on adversarial test sets. We develop a regularization strategy for model compression based on sample uncertainty.
  arXiv  Detail & Related papers  (2021-10-16T00:20:04Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)
• Analyzing and Mitigating JPEG Compression Defects in Deep Learning [69.04777875711646]
  We present a unified study of the effects of JPEG compression on a range of common tasks and datasets. We show that there is a significant penalty on common performance metrics for high compression.
  arXiv  Detail & Related papers  (2020-11-17T20:32:57Z)
• Neural Network Compression Via Sparse Optimization [23.184290795230897]
  We propose a model compression framework based on the recent progress on sparse optimization. We achieve up to 7.2 and 2.9 times FLOPs reduction with the same level of evaluation of accuracy on VGG16 for CIFAR10 and ResNet50 for ImageNet.
  arXiv  Detail & Related papers  (2020-11-10T03:03:55Z)
• GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework [94.26938614206689]
  We propose the first unified optimization framework combining multiple compression means for GAN compression, dubbed GAN Slimming. We apply GS to compress CartoonGAN, a state-of-the-art style transfer network, by up to 47 times, with minimal visual quality degradation.
  arXiv  Detail & Related papers  (2020-08-25T14:39:42Z)
• End-to-end Learning of Compressible Features [35.40108701875527]
  Pre-trained convolutional neural networks (CNNs) are powerful off-the-shelf feature generators. CNNs are powerful off-the-shelf feature generators and have been shown to perform very well on a variety of tasks. Unfortunately, the generated features are high dimensional and expensive to store. We propose a learned method that jointly optimize for compressibility along with the task objective.
  arXiv  Detail & Related papers  (2020-07-23T05:17:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.