Application Specific Compression of Deep Learning Models

• URL: http://arxiv.org/abs/2409.05368v1
• Date: Mon, 9 Sep 2024 06:55:38 GMT
• Title: Application Specific Compression of Deep Learning Models
• Authors: Rohit Raj Rai, Angana Borah, Amit Awekar,
• Abstract summary: Large Deep Learning models are compressed and deployed for specific applications. Our goal is to customize the model compression process to create a compressed model that will perform better for the target application. We have experimented with the BERT family of models for three applications: Extractive QA, Natural Language Inference, and Paraphrase Identification.
• Score: 0.8875650122536799
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Deep Learning models are compressed and deployed for specific applications. However, current Deep Learning model compression methods do not utilize the information about the target application. As a result, the compressed models are application agnostic. Our goal is to customize the model compression process to create a compressed model that will perform better for the target application. Our method, Application Specific Compression (ASC), identifies and prunes components of the large Deep Learning model that are redundant specifically for the given target application. The intuition of our work is to prune the parts of the network that do not contribute significantly to updating the data representation for the given application. We have experimented with the BERT family of models for three applications: Extractive QA, Natural Language Inference, and Paraphrase Identification. We observe that customized compressed models created using ASC method perform better than existing model compression methods and off-the-shelf compressed models.

Related papers

• CompLeak: Deep Learning Model Compression Exacerbates Privacy Leakage [14.170572219170158]
  We propose CompLeak, the first privacy risk framework examining three widely used compression configurations.<n> CompLeak has three variants, given available access to the number of compressed models and original model.
  arXiv  Detail & Related papers  (2025-07-22T08:02:46Z)
• Compression-Aware One-Step Diffusion Model for JPEG Artifact Removal [56.307484956135355]
  CODiff is a compression-aware one-step diffusion model for JPEG artifact removal. We propose a dual learning strategy that combines explicit and implicit learning. Results demonstrate that CODiff surpasses recent leading methods in both quantitative and visual quality metrics.
  arXiv  Detail & Related papers  (2025-02-14T02:46:27Z)
• Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles [49.65811277223873]
  Style-Compress is a lightweight framework that adapts a smaller language model to compress prompts for a larger model on a new task without additional training. Our approach iteratively generates and selects effective compressed prompts as task-specific demonstrations through style variation and in-context learning. Style-Compress outperforms two baseline compression models in four tasks: original prompt reconstruction, text summarization, multi-hop QA, and CoT reasoning.
  arXiv  Detail & Related papers  (2024-10-17T21:35:49Z)
• A Survey on Transformer Compression [84.18094368700379]
  Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV) Model compression methods reduce the memory and computational cost of Transformer. This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to Transformer-based models.
  arXiv  Detail & Related papers  (2024-02-05T12:16:28Z)
• Activations and Gradients Compression for Model-Parallel Training [85.99744701008802]
  We study how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We find that gradients require milder compression rates than activations. Experiments also show that models trained with TopK perform well only when compression is also applied during inference.
  arXiv  Detail & Related papers  (2024-01-15T15:54:54Z)
• Backdoor Attacks Against Deep Image Compression via Adaptive Frequency Trigger [106.10954454667757]
  We present a novel backdoor attack with multiple triggers against learned image compression models. Motivated by the widely used discrete cosine transform (DCT) in existing compression systems and standards, we propose a frequency-based trigger injection model.
  arXiv  Detail & Related papers  (2023-02-28T15:39:31Z)
• Deep learning model compression using network sensitivity and gradients [3.52359746858894]
  We present model compression algorithms for both non-retraining and retraining conditions. In the first case, we propose the Bin & Quant algorithm for compression of the deep learning models using the sensitivity of the network parameters. In the second case, we propose our novel gradient-weighted k-means clustering algorithm (GWK)
  arXiv  Detail & Related papers  (2022-10-11T03:02:40Z)
• Structural Dropout for Model Width Compression [1.52292571922932]
  Existing ML models are known to be highly over-parametrized, and use significantly more resources than required for a given task. We propose a method that requires only a single training session for the original model and a set of compressed models. The proposed approach is a "structural" dropout that prunes all elements in the hidden state above a randomly chosen index, forcing the model to learn an importance ordering over its features.
  arXiv  Detail & Related papers  (2022-05-13T21:50:57Z)
• What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression [68.82486784654817]
  We study two popular model compression techniques including knowledge distillation and pruning. We show that compressed models are significantly less robust than their PLM counterparts on adversarial test sets. We develop a regularization strategy for model compression based on sample uncertainty.
  arXiv  Detail & Related papers  (2021-10-16T00:20:04Z)
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search [100.71365025972258]
  We propose NAS-BERT, an efficient method for BERT compression. NAS-BERT trains a big supernet on a search space and outputs multiple compressed models with adaptive sizes and latency. Experiments on GLUE and SQuAD benchmark datasets demonstrate that NAS-BERT can find lightweight models with better accuracy than previous approaches.
  arXiv  Detail & Related papers  (2021-05-30T07:20:27Z)
• Self-Supervised GAN Compression [32.21713098893454]
  We show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.
  arXiv  Detail & Related papers  (2020-07-03T04:18:54Z)
• A flexible, extensible software framework for model compression based on the LC algorithm [10.787390511207683]
  We propose a software framework that allows a user to compress a neural network or other machine learning model with minimal effort. The library is written in Python and PyTorch and available in Github.
  arXiv  Detail & Related papers  (2020-05-15T21:14:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.