Related papers: Data Efficiency for Large Recommendation Models

Data Efficiency for Large Recommendation Models

URL: http://arxiv.org/abs/2410.18111v2
Date: Fri, 25 Oct 2024 10:26:35 GMT
Title: Data Efficiency for Large Recommendation Models
Authors: Kshitij Jain, Jingru Xie, Kevin Regan, Cheng Chen, Jie Han, Steve Li, Zhuoshu Li, Todd Phillips, Myles Sussman, Matt Troup, Angel Yu, Jia Zhuo,
Abstract summary: Large recommendation models (LRMs) are fundamental to the multi-billion dollar online advertising industry. The massive scale of data directly impacts both computational costs and the speed at which new methods can be evaluated. This paper presents actionable principles and high-level frameworks to guide practitioners in optimizing training data requirements.
Score: 4.799343040337817
License:
Abstract: Large recommendation models (LRMs) are fundamental to the multi-billion dollar online advertising industry, processing massive datasets of hundreds of billions of examples before transitioning to continuous online training to adapt to rapidly changing user behavior. The massive scale of data directly impacts both computational costs and the speed at which new methods can be evaluated (R&D velocity). This paper presents actionable principles and high-level frameworks to guide practitioners in optimizing training data requirements. These strategies have been successfully deployed in Google's largest Ads CTR prediction models and are broadly applicable beyond LRMs. We outline the concept of data convergence, describe methods to accelerate this convergence, and finally, detail how to optimally balance training data volume with model size.

Related papers

Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws [59.03420759554073]
We introduce Adaptive Data Optimization (ADO), an algorithm that optimize data distributions in an online fashion, concurrent with model training. ADO does not require external knowledge, proxy models, or modifications to the model update. ADO uses per-domain scaling laws to estimate the learning potential of each domain during training and adjusts the data mixture accordingly.
arXiv Detail & Related papers (2024-10-15T17:47:44Z)
AutoScale: Automatic Prediction of Compute-optimal Data Composition for Training LLMs [61.13296177652599]
This paper demonstrates that the optimal composition of training data from different domains is scale-dependent. We introduce *AutoScale*, a novel, practical approach for optimizing data compositions at potentially large training data scales. Our evaluation on GPT-2 Large and BERT pre-training demonstrates *AutoScale*'s effectiveness in improving training convergence and downstream performance.
arXiv Detail & Related papers (2024-07-29T17:06:30Z)
Predicting the Skies: A Novel Model for Flight-Level Passenger Traffic Forecasting [0.0]
This study introduces a novel, multimodal deep learning approach to the challenge of predicting flight-level passenger traffic. Our model ingests historical traffic data, fare closure information, and seasonality attributes specific to each flight. Our model demonstrates an approximate 33% improvement in Mean Squared Error compared to traditional benchmarks.
arXiv Detail & Related papers (2024-01-07T06:51:26Z)
Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation [59.899714450049494]
offline pre-training can produce sub-optimal policies and lead to degraded online reinforcement learning performance. We propose a model-based data augmentation strategy to maximize the benefits of offline reinforcement learning pre-training and reduce the scale of data needed to be effective.
arXiv Detail & Related papers (2023-12-15T14:49:41Z)
Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders [63.28408887247742]
We study whether training procedures can be improved to yield better generalization capabilities in the resulting models. We recommend a simple recipe for training dense encoders: Train on MSMARCO with parameter-efficient methods, such as LoRA, and opt for using in-batch negatives unless given well-constructed hard negatives.
arXiv Detail & Related papers (2023-11-16T10:42:58Z)
Data Augmentation Strategies for Improving Sequential Recommender Systems [7.986899327513767]
Sequential recommender systems have recently achieved significant performance improvements with the exploitation of deep learning (DL) based methods. We propose a set of data augmentation strategies, all of which transform original item sequences in the way of direct corruption. Experiments on the latest DL-based model show that applying data augmentation can help the model generalize better.
arXiv Detail & Related papers (2022-03-26T09:58:14Z)
Learning Distributionally Robust Models at Scale via Composite Optimization [45.47760229170775]
We show how different variants of DRO are simply instances of a finite-sum composite optimization for which we provide scalable methods. We also provide empirical results that demonstrate the effectiveness of our proposed algorithm with respect to the prior art in order to learn robust models from very large datasets.
arXiv Detail & Related papers (2022-03-17T20:47:42Z)
On Effective Scheduling of Model-based Reinforcement Learning [53.027698625496015]
We propose a framework named AutoMBPO to automatically schedule the real data ratio. In this paper, we first theoretically analyze the role of real data in policy training, which suggests that gradually increasing the ratio of real data yields better performance.
arXiv Detail & Related papers (2021-11-16T15:24:59Z)
Regularizing Generative Adversarial Networks under Limited Data [88.57330330305535]
This work proposes a regularization approach for training robust GAN models on limited data. We show a connection between the regularized loss and an f-divergence called LeCam-divergence, which we find is more robust under limited training data.
arXiv Detail & Related papers (2021-04-07T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.