A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation

• URL: http://arxiv.org/abs/2207.07483v1
• Date: Fri, 15 Jul 2022 14:09:04 GMT
• Title: A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation
• Authors: Aleksandr Petrov and Craig Macdonald
• Abstract summary: BERT4Rec is an effective model for sequential recommendation based on the Transformer architecture. We show that BERT4Rec results are not consistent within these publications. We propose our own implementation of BERT4Rec based on the Hugging Face Transformers library.
• Score: 91.02268704681124
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: BERT4Rec is an effective model for sequential recommendation based on the Transformer architecture. In the original publication, BERT4Rec claimed superiority over other available sequential recommendation approaches (e.g. SASRec), and it is now frequently being used as a state-of-the art baseline for sequential recommendations. However, not all subsequent publications confirmed this result and proposed other models that were shown to outperform BERT4Rec in effectiveness. In this paper we systematically review all publications that compare BERT4Rec with another popular Transformer-based model, namely SASRec, and show that BERT4Rec results are not consistent within these publications. To understand the reasons behind this inconsistency, we analyse the available implementations of BERT4Rec and show that we fail to reproduce results of the original BERT4Rec publication when using their default configuration parameters. However, we are able to replicate the reported results with the original code if training for a much longer amount of time (up to 30x) compared to the default configuration. We also propose our own implementation of BERT4Rec based on the Hugging Face Transformers library, which we demonstrate replicates the originally reported results on 3 out 4 datasets, while requiring up to 95% less training time to converge. Overall, from our systematic review and detailed experiments, we conclude that BERT4Rec does indeed exhibit state-of-the-art effectiveness for sequential recommendation, but only when trained for a sufficient amount of time. Additionally, we show that our implementation can further benefit from adapting other Transformer architectures that are available in the Hugging Face Transformers library (e.g. using disentangled attention, as provided by DeBERTa, or larger hidden layer size cf. ALBERT).

Related papers

• SetBERT: Enhancing Retrieval Performance for Boolean Logic and Set Operation Queries [0.8192907805418583]
  We introduce SetBERT, a fine-tuned BERT-based model to enhance query embeddings for set operations and Boolean logic queries. Our experiments reveal that SetBERT-base not only significantly outperforms BERT-base but also achieves performance comparable to the much larger BERT-large model.
  arXiv  Detail & Related papers  (2024-06-25T05:14:54Z)
• Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec? [1.223779595809275]
  Two state-of-the-art baselines are Transformer-based models SASRec and BERT4Rec. In most of the publications, BERT4Rec achieves better performance than SASRec. We show that SASRec could be effectively trained with negative sampling and still outperform BERT4Rec, but the number of negative examples should be much larger than one.
  arXiv  Detail & Related papers  (2023-09-14T11:07:10Z)
• Multi-BERT for Embeddings for Recommendation System [0.0]
  We propose a novel approach for generating document embeddings using a combination of Sentence-BERT and RoBERTa. Our approach treats sentences as tokens and generates embeddings for them, allowing the model to capture both intra-sentence and inter-sentence relations within a document. We evaluate our model on a book recommendation task and demonstrate its effectiveness in generating more semantically rich and accurate document embeddings.
  arXiv  Detail & Related papers  (2023-08-24T19:36:05Z)
• gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling [67.71952251641545]
  We show that models trained with negative sampling tend to overestimate the probabilities of positive interactions. We propose a novel Generalised Binary Cross-Entropy Loss function (gBCE) and theoretically prove that it can mitigate overconfidence. We show through detailed experiments on three datasets that gSASRec does not exhibit the overconfidence problem.
  arXiv  Detail & Related papers  (2023-08-14T14:56:40Z)
• Improving Sequential Recommendation Models with an Enhanced Loss Function [9.573139673704766]
  We develop an improved loss function for sequential recommendation models. We conduct experiments on two influential open-source libraries. We reproduce the results of the BERT4Rec model on the Beauty dataset.
  arXiv  Detail & Related papers  (2023-01-03T07:18:54Z)
• Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED [60.39125850987604]
  We show that a textit-revise scheme results in false negative samples and an obvious bias towards popular entities and relations. The relabeled dataset is released to serve as a more reliable test set of document RE models.
  arXiv  Detail & Related papers  (2022-04-17T11:29:01Z)
• BiBERT: Accurate Fully Binarized BERT [69.35727280997617]
  BiBERT is an accurate fully binarized BERT to eliminate the performance bottlenecks. Our method yields impressive 56.3 times and 31.2 times saving on FLOPs and model size.
  arXiv  Detail & Related papers  (2022-03-12T09:46:13Z)
• The MultiBERTs: BERT Reproductions for Robustness Analysis [86.29162676103385]
  Re-running pretraining can lead to substantially different conclusions about performance. We introduce MultiBERTs: a set of 25 BERT-base checkpoints. The aim is to enable researchers to draw robust and statistically justified conclusions about pretraining procedures.
  arXiv  Detail & Related papers  (2021-06-30T15:56:44Z)
• TernaryBERT: Distillation-aware Ultra-low Bit BERT [53.06741585060951]
  We propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods.
  arXiv  Detail & Related papers  (2020-09-27T10:17:28Z)
• Conformer-Kernel with Query Term Independence for Document Retrieval [32.36908635150144]
  The Transformer- Kernel (TK) model has demonstrated strong reranking performance on the TREC Deep Learning benchmark. We extend the TK architecture to the full retrieval setting by incorporating the query term independence assumption. We show that the Conformer's GPU memory requirement scales linearly with input sequence length, making it a more viable option when ranking long documents.
  arXiv  Detail & Related papers  (2020-07-20T19:47:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.