ID-centric Pre-training for Recommendation

• URL: http://arxiv.org/abs/2405.03562v2
• Date: Tue, 07 May 2024 06:29:25 GMT
• Title: ID-centric Pre-training for Recommendation
• Authors: Yiqing Wu, Ruobing Xie, Zhao Zhang, Fuzhen Zhuang, Xu Zhang, Leyu Lin, Zhanhui Kang, Yongjun Xu,
• Abstract summary: ID embeddings are challenging to be transferred to new domains. behavioral information in ID embeddings is still verified to be dominating in PLM-based recommendation models. We propose a novel ID-centric recommendation pre-training paradigm (IDP), which directly transfers informative ID embeddings learned in pre-training domains to item representations in new domains.
• Score: 51.72177873832969
• License:
• Abstract: Classical sequential recommendation models generally adopt ID embeddings to store knowledge learned from user historical behaviors and represent items. However, these unique IDs are challenging to be transferred to new domains. With the thriving of pre-trained language model (PLM), some pioneer works adopt PLM for pre-trained recommendation, where modality information (e.g., text) is considered universal across domains via PLM. Unfortunately, the behavioral information in ID embeddings is still verified to be dominating in PLM-based recommendation models compared to modality information and thus limits these models' performance. In this work, we propose a novel ID-centric recommendation pre-training paradigm (IDP), which directly transfers informative ID embeddings learned in pre-training domains to item representations in new domains. Specifically, in pre-training stage, besides the ID-based sequential model for recommendation, we also build a Cross-domain ID-matcher (CDIM) learned by both behavioral and modality information. In the tuning stage, modality information of new domain items is regarded as a cross-domain bridge built by CDIM. We first leverage the textual information of downstream domain items to retrieve behaviorally and semantically similar items from pre-training domains using CDIM. Next, these retrieved pre-trained ID embeddings, rather than certain textual embeddings, are directly adopted to generate downstream new items' embeddings. Through extensive experiments on real-world datasets, both in cold and warm settings, we demonstrate that our proposed model significantly outperforms all baselines. Codes will be released upon acceptance.

Related papers

• Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning [64.1745161657794]
  Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. Recent advances in pre-trained models provide a solid foundation for DIL. However, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. We propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge.
  arXiv  Detail & Related papers  (2024-10-01T17:58:06Z)
• Exploring User Retrieval Integration towards Large Language Models for Cross-Domain Sequential Recommendation [66.72195610471624]
  Cross-Domain Sequential Recommendation aims to mine and transfer users' sequential preferences across different domains. We propose a novel framework named URLLM, which aims to improve the CDSR performance by exploring the User Retrieval approach.
  arXiv  Detail & Related papers  (2024-06-05T09:19:54Z)
• Adapting to Distribution Shift by Visual Domain Prompt Generation [34.19066857066073]
  We adapt a model at test-time using a few unlabeled data to address distribution shifts. We build a knowledge bank to learn the transferable knowledge from source domains. The proposed method outperforms previous work on 5 large-scale benchmarks including WILDS and DomainNet.
  arXiv  Detail & Related papers  (2024-05-05T02:44:04Z)
• Quality > Quantity: Synthetic Corpora from Foundation Models for Closed-Domain Extractive Question Answering [35.38140071573828]
  We study extractive question answering within closed domains and introduce the concept of targeted pre-training. Our proposed framework uses Galactica to generate synthetic, targeted'' corpora that align with specific writing styles and topics.
  arXiv  Detail & Related papers  (2023-10-25T20:48:16Z)
• Online Unsupervised Domain Adaptation for Person Re-identification [4.48123023008698]
  We present a new yet practical online setting for Unsupervised Domain Adaptation for person Re-ID. We adapt and evaluate the state-of-the-art UDA algorithms on this new online setting using the well-known Market-1501, Duke, and MSMT17 benchmarks.
  arXiv  Detail & Related papers  (2022-05-09T15:36:08Z)
• Unified Instance and Knowledge Alignment Pretraining for Aspect-based Sentiment Analysis [96.53859361560505]
  Aspect-based Sentiment Analysis (ABSA) aims to determine the sentiment polarity towards an aspect. There always exists severe domain shift between the pretraining and downstream ABSA datasets. We introduce a unified alignment pretraining framework into the vanilla pretrain-finetune pipeline.
  arXiv  Detail & Related papers  (2021-10-26T04:03:45Z)
• Unsupervised Domain Adaptive Learning via Synthetic Data for Person Re-identification [101.1886788396803]
  Person re-identification (re-ID) has gained more and more attention due to its widespread applications in video surveillance. Unfortunately, the mainstream deep learning methods still need a large quantity of labeled data to train models. In this paper, we develop a data collector to automatically generate synthetic re-ID samples in a computer game, and construct a data labeler to simultaneously annotate them.
  arXiv  Detail & Related papers  (2021-09-12T15:51:41Z)
• Multi-Domain Adversarial Feature Generalization for Person Re-Identification [52.835955258959785]
  We propose a multi-dataset feature generalization network (MMFA-AAE) It is capable of learning a universal domain-invariant feature representation from multiple labeled datasets and generalizing it to unseen' camera systems. It also surpasses many state-of-the-art supervised methods and unsupervised domain adaptation methods by a large margin.
  arXiv  Detail & Related papers  (2020-11-25T08:03:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.