Mixed-Precision Embeddings for Large-Scale Recommendation Models

• URL: http://arxiv.org/abs/2409.20305v2
• Date: Thu, 17 Oct 2024 13:19:08 GMT
• Title: Mixed-Precision Embeddings for Large-Scale Recommendation Models
• Authors: Shiwei Li, Zhuoqi Hu, Xing Tang, Haozhao Wang, Shijie Xu, Weihong Luo, Yuhua Li, Xiuqiang He, Ruixuan Li,
• Abstract summary: Mixed-Precision Embeddings (MPE) is a novel embedding compression method. MPE achieves about 200x compression on the Criteo dataset without comprising the prediction accuracy.
• Score: 19.93156309493436
• License:
• Abstract: Embedding techniques have become essential components of large databases in the deep learning era. By encoding discrete entities, such as words, items, or graph nodes, into continuous vector spaces, embeddings facilitate more efficient storage, retrieval, and processing in large databases. Especially in the domain of recommender systems, millions of categorical features are encoded as unique embedding vectors, which facilitates the modeling of similarities and interactions among features. However, numerous embedding vectors can result in significant storage overhead. In this paper, we aim to compress the embedding table through quantization techniques. Given that features vary in importance levels, we seek to identify an appropriate precision for each feature to balance model accuracy and memory usage. To this end, we propose a novel embedding compression method, termed Mixed-Precision Embeddings (MPE). Specifically, to reduce the size of the search space, we first group features by frequency and then search precision for each feature group. MPE further learns the probability distribution over precision levels for each feature group, which can be used to identify the most suitable precision with a specially designed sampling strategy. Extensive experiments on three public datasets demonstrate that MPE significantly outperforms existing embedding compression methods. Remarkably, MPE achieves about 200x compression on the Criteo dataset without comprising the prediction accuracy.

Related papers

• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
  Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
  arXiv  Detail & Related papers  (2024-05-09T11:49:05Z)
• RGM: A Robust Generalizable Matching Model [49.60975442871967]
  We propose a deep model for sparse and dense matching, termed RGM (Robust Generalist Matching) To narrow the gap between synthetic training samples and real-world scenarios, we build a new, large-scale dataset with sparse correspondence ground truth. We are able to mix up various dense and sparse matching datasets, significantly improving the training diversity.
  arXiv  Detail & Related papers  (2023-10-18T07:30:08Z)
• Binary Embedding-based Retrieval at Tencent [30.44247353560061]
  Large-scale embedding-based retrieval (EBR) is the cornerstone of search-related industrial applications. We propose a binary embedding-based retrieval engine equipped with a recurrent binarization algorithm that enables customized bits per dimension. We successfully employed the introduced BEBR to Tencent products, including Sogou, Tencent Video, QQ World, etc.
  arXiv  Detail & Related papers  (2023-02-17T06:10:02Z)
• Variational Factorization Machines for Preference Elicitation in Large-Scale Recommender Systems [17.050774091903552]
  We propose a variational formulation of factorization machines (FMs) that can be easily optimized using standard mini-batch descent gradient. Our algorithm learns an approximate posterior distribution over the user and item parameters, which leads to confidence intervals over the predictions. We show, using several datasets, that it has comparable or better performance than existing methods in terms of prediction accuracy.
  arXiv  Detail & Related papers  (2022-12-20T00:06:28Z)
• CloudAttention: Efficient Multi-Scale Attention Scheme For 3D Point Cloud Learning [81.85951026033787]
  We set transformers in this work and incorporate them into a hierarchical framework for shape classification and part and scene segmentation. We also compute efficient and dynamic global cross attentions by leveraging sampling and grouping at each iteration. The proposed hierarchical model achieves state-of-the-art shape classification in mean accuracy and yields results on par with the previous segmentation methods.
  arXiv  Detail & Related papers  (2022-07-31T21:39:15Z)
• Automatic Mixed-Precision Quantization Search of BERT [62.65905462141319]
  Pre-trained language models such as BERT have shown remarkable effectiveness in various natural language processing tasks. These models usually contain millions of parameters, which prevents them from practical deployment on resource-constrained devices. We propose an automatic mixed-precision quantization framework designed for BERT that can simultaneously conduct quantization and pruning in a subgroup-wise level.
  arXiv  Detail & Related papers  (2021-12-30T06:32:47Z)
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty [58.144520501201995]
  Bi-Lipschitz regularization of neural network layers preserve relative distances between data instances in the feature spaces of each layer. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. We also propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution.
  arXiv  Detail & Related papers  (2021-10-12T22:04:19Z)
• Generalizable Mixed-Precision Quantization via Attribution Rank Preservation [90.26603048354575]
  We propose a generalizable mixed-precision quantization (GMPQ) method for efficient inference. Our method obtains competitive accuracy-complexity trade-off compared with the state-of-the-art mixed-precision networks.
  arXiv  Detail & Related papers  (2021-08-05T16:41:57Z)
• End-to-end Learning of Compressible Features [35.40108701875527]
  Pre-trained convolutional neural networks (CNNs) are powerful off-the-shelf feature generators. CNNs are powerful off-the-shelf feature generators and have been shown to perform very well on a variety of tasks. Unfortunately, the generated features are high dimensional and expensive to store. We propose a learned method that jointly optimize for compressibility along with the task objective.
  arXiv  Detail & Related papers  (2020-07-23T05:17:33Z)
• Multi-Person Pose Estimation with Enhanced Feature Aggregation and Selection [33.15192824888279]
  We propose a novel Enhanced Feature Aggregation and Selection network (EFASNet) for multi-person 2D human pose estimation. Our method can well handle crowded, cluttered and occluded scenes. Comprehensive experiments demonstrate that the proposed approach outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-03-20T08:33:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.