BARS-CTR: Open Benchmarking for Click-Through Rate Prediction

• URL: http://arxiv.org/abs/2009.05794v5
• Date: Thu, 30 Nov 2023 02:01:41 GMT
• Title: BARS-CTR: Open Benchmarking for Click-Through Rate Prediction
• Authors: Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, Xiuqiang He
• Abstract summary: Click-through rate (CTR) prediction is a critical task for many applications. In recent years, CTR prediction has been widely studied in both academia and industry. There is still a lack of standardized benchmarks and uniform evaluation protocols for CTR prediction research.
• Score: 30.000261789268063
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Click-through rate (CTR) prediction is a critical task for many applications, as its accuracy has a direct impact on user experience and platform revenue. In recent years, CTR prediction has been widely studied in both academia and industry, resulting in a wide variety of CTR prediction models. Unfortunately, there is still a lack of standardized benchmarks and uniform evaluation protocols for CTR prediction research. This leads to non-reproducible or even inconsistent experimental results among existing studies, which largely limits the practical value and potential impact of their research. In this work, we aim to perform open benchmarking for CTR prediction and present a rigorous comparison of different models in a reproducible manner. To this end, we ran over 7,000 experiments for more than 12,000 GPU hours in total to re-evaluate 24 existing models on multiple datasets and settings. Surprisingly, our experiments show that with sufficient hyper-parameter search and model tuning, many deep models have smaller differences than expected. The results also reveal that making real progress on the modeling of CTR prediction is indeed a very challenging research task. We believe that our benchmarking work could not only allow researchers to gauge the effectiveness of new models conveniently but also make them fairly compare with the state of the arts. We have publicly released the benchmarking code, evaluation protocols, and hyper-parameter settings of our work to promote reproducible research in this field.

Related papers

• Benchmarking for Deep Uplift Modeling in Online Marketing [17.70084353772874]
  Deep uplift modeling (DUM) as a promising technique has attracted increased research from academia and industry. Current DUM still lacks some standardized benchmarks and unified evaluation protocols. We provide an open benchmark for DUM and present comparison results of existing models in a reproducible and uniform manner.
  arXiv  Detail & Related papers  (2024-06-01T07:23:37Z)
• Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study [61.64685376882383]
  Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models. This paper investigates the robustness of existing CLTR models in complex and diverse situations. We find that the DLA models and IPS-DCM show better robustness under various simulation settings than IPS-PBM and PRS with offline propensity estimation.
  arXiv  Detail & Related papers  (2024-04-04T10:54:38Z)
• Predictive Churn with the Set of Good Models [64.05949860750235]
  We study the effect of conflicting predictions over the set of near-optimal machine learning models. We present theoretical results on the expected churn between models within the Rashomon set. We show how our approach can be used to better anticipate, reduce, and avoid churn in consumer-facing applications.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• Small Effect Sizes in Malware Detection? Make Harder Train/Test Splits! [51.668411293817464]
  Industry practitioners care about small improvements in malware detection accuracy because their models are deployed to hundreds of millions of machines. Academic research is often restrained to public datasets on the order of ten thousand samples. We devise an approach to generate a benchmark of difficulty from a pool of available samples.
  arXiv  Detail & Related papers  (2023-12-25T21:25:55Z)
• MRCLens: an MRC Dataset Bias Detection Toolkit [82.44296974850639]
  We introduce MRCLens, a toolkit that detects whether biases exist before users train the full model. For the convenience of introducing the toolkit, we also provide a categorization of common biases in MRC.
  arXiv  Detail & Related papers  (2022-07-18T21:05:39Z)
• A Thorough Examination on Zero-shot Dense Retrieval [84.70868940598143]
  We present the first thorough examination of the zero-shot capability of dense retrieval (DR) models. We discuss the effect of several key factors related to source training set, analyze the potential bias from the target dataset, and review and compare existing zero-shot DR models.
  arXiv  Detail & Related papers  (2022-04-27T07:59:07Z)
• Click-Through Rate Prediction in Online Advertising: A Literature Review [0.0]
  We make a systematic literature review on state-of-the-art and latest CTR prediction research. We give a classification of state-of-the-art CTR prediction models in the extant literature. We identify current research trends, main challenges and potential future directions worthy of further explorations.
  arXiv  Detail & Related papers  (2022-02-22T01:05:38Z)
• Adversarial Gradient Driven Exploration for Deep Click-Through Rate Prediction [39.61776002290324]
  We propose a novel exploration method called textbfAdrial textbfGradientversa Driven textbfExploration (AGE) AGE simulates the gradient updating process, which can approximate the influence of the samples of to-be-explored items for the model. The effectiveness of our approach was demonstrated on an open-access academic dataset.
  arXiv  Detail & Related papers  (2021-12-21T12:13:07Z)
• Looking at CTR Prediction Again: Is Attention All You Need? [4.873362301533825]
  Click-through rate (CTR) prediction is a critical problem in web search, recommendation systems and online advertisement displaying. We use the discrete choice model in economics to redefine the CTR prediction problem, and propose a general neural network framework built on self-attention mechanism. It is found that most existing CTR prediction models align with our proposed general framework.
  arXiv  Detail & Related papers  (2021-05-12T10:27:14Z)
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning [109.74041512359476]
  We study a number of design decisions for the predictive model in visual MBRL algorithms. We find that a range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance. We show how this phenomenon is related to exploration and how some of the lower-scoring models on standard benchmarks will perform the same as the best-performing models when trained on the same training data.
  arXiv  Detail & Related papers  (2020-12-08T18:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.