Enhancing CTR Prediction through Sequential Recommendation Pre-training: Introducing the SRP4CTR Framework

• URL: http://arxiv.org/abs/2407.19658v1
• Date: Mon, 29 Jul 2024 02:49:11 GMT
• Title: Enhancing CTR Prediction through Sequential Recommendation Pre-training: Introducing the SRP4CTR Framework
• Authors: Ruidong Han, Qianzhong Li, He Jiang, Rui Li, Yurou Zhao, Xiang Li, Wei Lin,
• Abstract summary: We propose a Sequential Recommendation Pre-training framework for Click-Through Rate (CTR) prediction (SRP4CTR) We discuss the impact of introducing pre-trained models on inference costs. Subsequently, we introduce a pre-trained method to encode sequence side information concurrently. We develop a querying transformer technique to facilitate the knowledge transfer from the pre-trained model to industrial CTR models.
• Score: 13.574487867743773
• License:
• Abstract: Understanding user interests is crucial for Click-Through Rate (CTR) prediction tasks. In sequential recommendation, pre-training from user historical behaviors through self-supervised learning can better comprehend user dynamic preferences, presenting the potential for direct integration with CTR tasks. Previous methods have integrated pre-trained models into downstream tasks with the sole purpose of extracting semantic information or well-represented user features, which are then incorporated as new features. However, these approaches tend to ignore the additional inference costs to the downstream tasks, and they do not consider how to transfer the effective information from the pre-trained models for specific estimated items in CTR prediction. In this paper, we propose a Sequential Recommendation Pre-training framework for CTR prediction (SRP4CTR) to tackle the above problems. Initially, we discuss the impact of introducing pre-trained models on inference costs. Subsequently, we introduced a pre-trained method to encode sequence side information concurrently.During the fine-tuning process, we incorporate a cross-attention block to establish a bridge between estimated items and the pre-trained model at a low cost. Moreover, we develop a querying transformer technique to facilitate the knowledge transfer from the pre-trained model to industrial CTR models. Offline and online experiments show that our method outperforms previous baseline models.

Related papers

• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Initialization Matters for Adversarial Transfer Learning [61.89451332757625]
  We discover the necessity of an adversarially robust pretrained model. We propose Robust Linear Initialization (RoLI) for adversarial finetuning, which initializes the linear head with the weights obtained by adversarial linear probing. Across five different image classification datasets, we demonstrate the effectiveness of RoLI and achieve new state-of-the-art results.
  arXiv  Detail & Related papers  (2023-12-10T00:51:05Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• Effective and Efficient Training for Sequential Recommendation using Recency Sampling [91.02268704681124]
  We propose a novel Recency-based Sampling of Sequences training objective. We show that the models enhanced with our method can achieve performances exceeding or very close to stateof-the-art BERT4Rec.
  arXiv  Detail & Related papers  (2022-07-06T13:06:31Z)
• Continual Learning for CTR Prediction: A Hybrid Approach [37.668467137218286]
  We propose COLF, a hybrid COntinual Learning Framework for CTR prediction. COLF has a memory-based modular architecture that is designed to adapt, learn and give predictions continuously. Empirical evaluations on click log collected from a major shopping app in China demonstrate our method's superiority over existing methods.
  arXiv  Detail & Related papers  (2022-01-18T11:30:57Z)
• Self-Supervised Pretraining Improves Self-Supervised Pretraining [83.1423204498361]
  Self-supervised pretraining requires expensive and lengthy computation, large amounts of data, and is sensitive to data augmentation. This paper explores Hierarchical PreTraining (HPT), which decreases convergence time and improves accuracy by initializing the pretraining process with an existing pretrained model. We show HPT converges up to 80x faster, improves accuracy across tasks, and improves the robustness of the self-supervised pretraining process to changes in the image augmentation policy or amount of pretraining data.
  arXiv  Detail & Related papers  (2021-03-23T17:37:51Z)
• Deep Ensembles for Low-Data Transfer Learning [21.578470914935938]
  We study different ways of creating ensembles from pre-trained models. We show that the nature of pre-training itself is a performant source of diversity. We propose a practical algorithm that efficiently identifies a subset of pre-trained models for any downstream dataset.
  arXiv  Detail & Related papers  (2020-10-14T07:59:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.