Pre-train, Align, and Disentangle: Empowering Sequential Recommendation with Large Language Models

• URL: http://arxiv.org/abs/2412.04107v1
• Date: Thu, 05 Dec 2024 12:17:56 GMT
• Title: Pre-train, Align, and Disentangle: Empowering Sequential Recommendation with Large Language Models
• Authors: Yuhao Wang, Junwei Pan, Xiangyu Zhao, Pengyue Jia, Wanyu Wang, Yuan Wang, Yue Liu, Dapeng Liu, Jie Jiang,
• Abstract summary: Sequential recommendation (SR) aims to model the sequential dependencies in users' historical interactions to better capture their evolving interests. Existing SR approaches rely on collaborative data, which leads to limitations such as the cold-start problem and sub-optimal performance. We propose a novel Pre-train, Align, and Disentangle (PAD) paradigm to empower recommendation models with large language models.
• Score: 38.15316444108154
• License:
• Abstract: Sequential recommendation (SR) aims to model the sequential dependencies in users' historical interactions to better capture their evolving interests. However, existing SR approaches primarily rely on collaborative data, which leads to limitations such as the cold-start problem and sub-optimal performance. Meanwhile, despite the success of large language models (LLMs), their application in industrial recommender systems is hindered by high inference latency, inability to capture all distribution statistics, and catastrophic forgetting. To this end, we propose a novel Pre-train, Align, and Disentangle (PAD) paradigm to empower recommendation models with LLMs. Specifically, we first pre-train both the SR and LLM models to get collaborative and textual embeddings. Next, a characteristic recommendation-anchored alignment loss is proposed using multi-kernel maximum mean discrepancy with Gaussian kernels. Finally, a triple-experts architecture, consisting aligned and modality-specific experts with disentangled embeddings, is fine-tuned in a frequency-aware manner. Experiments conducted on three public datasets demonstrate the effectiveness of PAD, showing significant improvements and compatibility with various SR backbone models, especially on cold items. The implementation code and datasets will be publicly available.

Related papers

• Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy [104.48511402784763]
  Performance Law for SR models aims to theoretically investigate and model the relationship between model performance and data quality. We propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics.
  arXiv  Detail & Related papers  (2024-11-30T10:56:30Z)
• Behavior-Dependent Linear Recurrent Units for Efficient Sequential Recommendation [18.75561256311228]
  RecBLR is an Efficient Sequential Recommendation Model based on Behavior-Dependent Linear Recurrent Units. Our model significantly enhances user behavior modeling and recommendation performance.
  arXiv  Detail & Related papers  (2024-06-18T13:06:58Z)
• Diffusion Augmentation for Sequential Recommendation [47.43402785097255]
  We propose a Diffusion Augmentation for Sequential Recommendation (DiffuASR) for a higher quality generation. The augmented dataset by DiffuASR can be used to train the sequential recommendation models directly, free from complex training procedures. We conduct extensive experiments on three real-world datasets with three sequential recommendation models.
  arXiv  Detail & Related papers  (2023-09-22T13:31:34Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)
• Contrastive Self-supervised Sequential Recommendation with Robust Augmentation [101.25762166231904]
  Sequential Recommendationdescribes a set of techniques to model dynamic user behavior in order to predict future interactions in sequential user data. Old and new issues remain, including data-sparsity and noisy data. We propose Contrastive Self-Supervised Learning for sequential Recommendation (CoSeRec)
  arXiv  Detail & Related papers  (2021-08-14T07:15:25Z)
• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization [104.87483578308526]
  We propose the model S3-Rec, which stands for Self-Supervised learning for Sequential Recommendation. For our task, we devise four auxiliary self-supervised objectives to learn the correlations among attribute, item, subsequence, and sequence. Extensive experiments conducted on six real-world datasets demonstrate the superiority of our proposed method over existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-08-18T11:44:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.