RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructure

• URL: http://arxiv.org/abs/2211.05239v4
• Date: Mon, 1 May 2023 19:37:39 GMT
• Title: RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructure
• Authors: Mark Zhao, Dhruv Choudhary, Devashish Tyagi, Ajay Somani, Max Kaplan, Sung-Han Lin, Sarunya Pumma, Jongsoo Park, Aarti Basant, Niket Agarwal, Carole-Jean Wu, Christos Kozyrakis
• Abstract summary: RecD addresses immense storage, preprocessing, and training overheads caused by feature duplication inherent in industry-scale DLRM training datasets. We show how RecD improves the training and preprocessing throughput and storage efficiency by up to 2.48x, 1.79x, and 3.71x, respectively, in an industry-scale DLRM training system.
• Score: 3.991664287163157
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present RecD (Recommendation Deduplication), a suite of end-to-end infrastructure optimizations across the Deep Learning Recommendation Model (DLRM) training pipeline. RecD addresses immense storage, preprocessing, and training overheads caused by feature duplication inherent in industry-scale DLRM training datasets. Feature duplication arises because DLRM datasets are generated from interactions. While each user session can generate multiple training samples, many features' values do not change across these samples. We demonstrate how RecD exploits this property, end-to-end, across a deployed training pipeline. RecD optimizes data generation pipelines to decrease dataset storage and preprocessing resource demands and to maximize duplication within a training batch. RecD introduces a new tensor format, InverseKeyedJaggedTensors (IKJTs), to deduplicate feature values in each batch. We show how DLRM model architectures can leverage IKJTs to drastically increase training throughput. RecD improves the training and preprocessing throughput and storage efficiency by up to 2.48x, 1.79x, and 3.71x, respectively, in an industry-scale DLRM training system.

Related papers

• Scaling Retrieval-Based Language Models with a Trillion-Token Datastore [85.4310806466002]
  We find that increasing the size of the datastore used by a retrieval-based LM monotonically improves language modeling and several downstream tasks without obvious saturation. By plotting compute-optimal scaling curves with varied datastore, model, and pretraining data sizes, we show that using larger datastores can significantly improve model performance for the same training compute budget.
  arXiv  Detail & Related papers  (2024-07-09T08:27:27Z)
• ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation [2.0181279529015925]
  ReCycle is a system designed for efficient training in the presence of failures. It exploits the inherent functional redundancy in distributed training systems. We show it achieves high training throughput under multiple failures.
  arXiv  Detail & Related papers  (2024-05-22T21:35:56Z)
• RA-DIT: Retrieval-Augmented Dual Instruction Tuning [90.98423540361946]
  Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option.
  arXiv  Detail & Related papers  (2023-10-02T17:16:26Z)
• InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models [3.7414278978078204]
  Deep learning-based recommender models (DLRMs) have become an essential component of many modern recommender systems. The systems challenges faced in this setting are unique; while typical deep learning training jobs are dominated by model execution, the most important factor in DLRM training performance is often online data ingestion.
  arXiv  Detail & Related papers  (2023-08-13T18:28:56Z)
• DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud [13.996191403653754]
  Deep learning models (DLRM) rely on large embedding tables to manage sparse features. Expanding such embedding tables can significantly enhance model performance, but at the cost of increased GPU/ CPU/memory usage. Tech companies have built extensive cloud-based services to accelerate training DLRM models at scale. We introduce DLRover-RM, an elastic training framework for DLRM to increase resource utilization and handle the instability of a cloud environment.
  arXiv  Detail & Related papers  (2023-04-04T02:13:46Z)
• Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines [77.45213180689952]
  Preprocessing pipelines in deep learning aim to provide sufficient data throughput to keep the training processes busy. We introduce a new perspective on efficiently preparing datasets for end-to-end deep learning pipelines. We obtain an increased throughput of 3x to 13x compared to an untuned system.
  arXiv  Detail & Related papers  (2022-02-17T14:31:58Z)
• Multi-Domain Joint Training for Person Re-Identification [51.73921349603597]
  Deep learning-based person Re-IDentification (ReID) often requires a large amount of training data to achieve good performance. It appears that collecting more training data from diverse environments tends to improve the ReID performance. We propose an approach called Domain-Camera-Sample Dynamic network (DCSD) whose parameters can be adaptive to various factors.
  arXiv  Detail & Related papers  (2022-01-06T09:20:59Z)
• Dynamic Network-Assisted D2D-Aided Coded Distributed Learning [59.29409589861241]
  We propose a novel device-to-device (D2D)-aided coded federated learning method (D2D-CFL) for load balancing across devices. We derive an optimal compression rate for achieving minimum processing time and establish its connection with the convergence time. Our proposed method is beneficial for real-time collaborative applications, where the users continuously generate training data.
  arXiv  Detail & Related papers  (2021-11-26T18:44:59Z)
• Understanding and Co-designing the Data Ingestion Pipeline for Industry-Scale RecSys Training [5.058493679956239]
  We present an extensive characterization of the data ingestion challenges for industry-scale recommendation model training. First, dataset storage requirements are massive and variable; exceeding local storage capacities. Secondly, reading and preprocessing data is computationally expensive, requiring substantially more compute, memory, and network resources than are available on trainers themselves. We introduce Data PreProcessing Service (DPP), a fully disaggregated preprocessing service that scales to hundreds of nodes, eliminating data stalls that can reduce training throughput by 56%.
  arXiv  Detail & Related papers  (2021-08-20T21:09:34Z)
• ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding [1.418033127602866]
  Deep-learning recommendation models (DLRMs) are widely deployed to serve personalized content to users. DLRMs are large in size due to their use of large embedding tables, and are trained by distributing the model across the memory of tens or hundreds of servers. Checkpointing is the primary approach used for fault tolerance in these systems, but incurs significant training-time overhead.
  arXiv  Detail & Related papers  (2021-04-05T16:16:19Z)
• Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism [56.78673028601739]
  We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT reduces communication by at least $100times$ and $20times$ during DP and MP, respectively. It improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.
  arXiv  Detail & Related papers  (2020-10-18T01:44:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.