Unified Low-rank Compression Framework for Click-through Rate Prediction

• URL: http://arxiv.org/abs/2405.18146v2
• Date: Tue, 11 Jun 2024 06:47:50 GMT
• Title: Unified Low-rank Compression Framework for Click-through Rate Prediction
• Authors: Hao Yu, Minghao Fu, Jiandong Ding, Yusheng Zhou, Jianxin Wu,
• Abstract summary: We propose a unified low-rank decomposition framework for compressing CTR prediction models. Our framework can achieve better performance than the original model. Our framework can be applied to embedding tables and layers in various CTR prediction models.
• Score: 15.813889566241539
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Deep Click-Through Rate (CTR) prediction models play an important role in modern industrial recommendation scenarios. However, high memory overhead and computational costs limit their deployment in resource-constrained environments. Low-rank approximation is an effective method for computer vision and natural language processing models, but its application in compressing CTR prediction models has been less explored. Due to the limited memory and computing resources, compression of CTR prediction models often confronts three fundamental challenges, i.e., (1). How to reduce the model sizes to adapt to edge devices? (2). How to speed up CTR prediction model inference? (3). How to retain the capabilities of original models after compression? Previous low-rank compression research mostly uses tensor decomposition, which can achieve a high parameter compression ratio, but brings in AUC degradation and additional computing overhead. To address these challenges, we propose a unified low-rank decomposition framework for compressing CTR prediction models. We find that even with the most classic matrix decomposition SVD method, our framework can achieve better performance than the original model. To further improve the effectiveness of our framework, we locally compress the output features instead of compressing the model weights. Our unified low-rank compression framework can be applied to embedding tables and MLP layers in various CTR prediction models. Extensive experiments on two academic datasets and one real industrial benchmark demonstrate that, with 3-5x model size reduction, our compressed models can achieve both faster inference and higher AUC than the uncompressed original models. Our code is at https://github.com/yuhao318/Atomic_Feature_Mimicking.

Related papers

• MoDeGPT: Modular Decomposition for Large Language Model Compression [59.361006801465344]
  This paper introduces textbfModular bfDecomposition (MoDeGPT), a novel structured compression framework. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions. Our experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods.
  arXiv  Detail & Related papers  (2024-08-19T01:30:14Z)
• Activations and Gradients Compression for Model-Parallel Training [85.99744701008802]
  We study how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We find that gradients require milder compression rates than activations. Experiments also show that models trained with TopK perform well only when compression is also applied during inference.
  arXiv  Detail & Related papers  (2024-01-15T15:54:54Z)
• Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression [8.591088380355252]
  We present Optimus-CC, a fast and scalable distributed training framework for large NLP models with aggressive communication compression. We propose techniques to avoid the model quality drop that comes from the compression. We demonstrate our solution on a GPU cluster, and achieve superior speedup from the baseline state-of-the-art solutions for distributed training without sacrificing the model quality.
  arXiv  Detail & Related papers  (2023-01-24T06:07:55Z)
• CrAM: A Compression-Aware Minimizer [103.29159003723815]
  We propose a new compression-aware minimizer dubbed CrAM that modifies the optimization step in a principled way. CrAM produces dense models that can be more accurate than the standard SGD/Adam-based baselines, but which are stable under weight pruning. CrAM can produce sparse models which perform well for transfer learning, and it also works for semi-structured 2:4 pruning patterns supported by GPU hardware.
  arXiv  Detail & Related papers  (2022-07-28T16:13:28Z)
• What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression [68.82486784654817]
  We study two popular model compression techniques including knowledge distillation and pruning. We show that compressed models are significantly less robust than their PLM counterparts on adversarial test sets. We develop a regularization strategy for model compression based on sample uncertainty.
  arXiv  Detail & Related papers  (2021-10-16T00:20:04Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)
• Overfitting for Fun and Profit: Instance-Adaptive Data Compression [20.764189960709164]
  Neural data compression has been shown to outperform classical methods in terms of $RD$ performance. In this paper we take this concept to the extreme, adapting the full model to a single video, and sending model updates along with the latent representation. We demonstrate that full-model adaptation improves $RD$ performance by 1 dB, with respect to encoder-only finetuning.
  arXiv  Detail & Related papers  (2021-01-21T15:58:58Z)
• Reliable Model Compression via Label-Preservation-Aware Loss Functions [14.368823297066276]
  We present a framework that uses a teacher-student learning paradigm to better preserve labels. We obtain a significant reduction of up to 4.1X in the number of mismatches between the compressed and reference models.
  arXiv  Detail & Related papers  (2020-12-03T00:00:41Z)
• Self-Supervised GAN Compression [32.21713098893454]
  We show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.
  arXiv  Detail & Related papers  (2020-07-03T04:18:54Z)
• A Generic Network Compression Framework for Sequential Recommender Systems [71.81962915192022]
  Sequential recommender systems (SRS) have become the key technology in capturing user's dynamic interests and generating high-quality recommendations. We propose a compressed sequential recommendation framework, termed as CpRec, where two generic model shrinking techniques are employed. By the extensive ablation studies, we demonstrate that the proposed CpRec can achieve up to 4$sim$8 times compression rates in real-world SRS datasets.
  arXiv  Detail & Related papers  (2020-04-21T08:40:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.