Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer

• URL: http://arxiv.org/abs/2105.00522v1
• Date: Sun, 2 May 2021 18:06:23 GMT
• Title: Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer
• Authors: Zhiwei Liu, Ziwei Fan, Yu Wang, Philip S. Yu
• Abstract summary: Sequential Recommendation characterizes the evolving patterns by modeling item sequences chronologically. Recent developments of transformer inspire the community to design effective sequence encoders. We introduce a new framework for textbfAugmenting textbfSequential textbfRecommendation with textbfPseudo-prior items(ASReP)
• Score: 61.818320703583126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sequential Recommendation characterizes the evolving patterns by modeling item sequences chronologically. The essential target of it is to capture the item transition correlations. The recent developments of transformer inspire the community to design effective sequence encoders, \textit{e.g.,} SASRec and BERT4Rec. However, we observe that these transformer-based models suffer from the cold-start issue, \textit{i.e.,} performing poorly for short sequences. Therefore, we propose to augment short sequences while still preserving original sequential correlations. We introduce a new framework for \textbf{A}ugmenting \textbf{S}equential \textbf{Re}commendation with \textbf{P}seudo-prior items~(ASReP). We firstly pre-train a transformer with sequences in a reverse direction to predict prior items. Then, we use this transformer to generate fabricated historical items at the beginning of short sequences. Finally, we fine-tune the transformer using these augmented sequences from the time order to predict the next item. Experiments on two real-world datasets verify the effectiveness of ASReP. The code is available on \url{https://github.com/DyGRec/ASReP}.

Related papers

• PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting [82.03373838627606]
  Self-attention mechanism in Transformer architecture requires positional embeddings to encode temporal order in time series prediction. We argue that this reliance on positional embeddings restricts the Transformer's ability to effectively represent temporal sequences. We present a model integrating PRE with a standard Transformer encoder, demonstrating state-of-the-art performance on various real-world datasets.
  arXiv  Detail & Related papers  (2024-08-20T01:56:07Z)
• An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention [23.610204672115195]
  We present pioneering investigations that reveal the low-pass filtering nature of self-attention in Sequential recommendation (SR) models. We propose a novel method calledBSARec, which injects an inductive bias by considering fine-grained sequential patterns. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models.
  arXiv  Detail & Related papers  (2023-12-16T05:23:08Z)
• GBT: Two-stage transformer framework for non-stationary time series forecasting [3.830797055092574]
  We propose GBT, a novel two-stage Transformer framework with Good Beginning. It decouples the prediction process of TSFT into two stages, including Auto-Regression stage and Self-Regression stage. Experiments on seven benchmark datasets demonstrate that GBT outperforms SOTA TSFTs with only canonical attention and convolution.
  arXiv  Detail & Related papers  (2023-07-17T07:55:21Z)
• Sequential Recommendation via Stochastic Self-Attention [68.52192964559829]
  Transformer-based approaches embed items as vectors and use dot-product self-attention to measure the relationship between items. We propose a novel textbfSTOchastic textbfSelf-textbfAttention(STOSA) to overcome these issues. We devise a novel Wasserstein Self-Attention module to characterize item-item position-wise relationships in sequences.
  arXiv  Detail & Related papers  (2022-01-16T12:38:45Z)
• Pose Transformers (POTR): Human Motion Prediction with Non-Autoregressive Transformers [24.36592204215444]
  We propose to leverage Transformer architectures for non-autoregressive human motion prediction. Our approach decodes elements in parallel from a query sequence, instead of conditioning on previous predictions. We show that despite its simplicity, our approach achieves competitive results in two public datasets.
  arXiv  Detail & Related papers  (2021-09-15T18:55:15Z)
• Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer [69.0621959845251]
  We propose a new framework Temporal Graph Sequential Recommender (TGSRec) upon our defined continuous-time bi-partite graph. TCT layer can simultaneously capture collaborative signals from both users and items, as well as considering temporal dynamics inside sequential patterns. Empirical results on five datasets show that TGSRec significantly outperforms other baselines.
  arXiv  Detail & Related papers  (2021-08-14T22:50:53Z)
• Contrastive Self-supervised Sequential Recommendation with Robust Augmentation [101.25762166231904]
  Sequential Recommendationdescribes a set of techniques to model dynamic user behavior in order to predict future interactions in sequential user data. Old and new issues remain, including data-sparsity and noisy data. We propose Contrastive Self-Supervised Learning for sequential Recommendation (CoSeRec)
  arXiv  Detail & Related papers  (2021-08-14T07:15:25Z)
• Finetuning Pretrained Transformers into RNNs [81.72974646901136]
  Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. A linear-complexity recurrent variant has proven well suited for autoregressive generation. This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
  arXiv  Detail & Related papers  (2021-03-24T10:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.