Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives

• URL: http://arxiv.org/abs/2512.01179v1
• Date: Mon, 01 Dec 2025 01:36:55 GMT
• Title: Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives
• Authors: Shan Gao, Yanwu Yang,
• Abstract summary: This research designs a unified architecture of CTR prediction benchmark (Bench-CTR) platform.<n>We construct a comprehensive system of evaluation protocols encompassing real-world and synthetic datasets.
• Score: 12.546199858068519
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This research designs a unified architecture of CTR prediction benchmark (Bench-CTR) platform that offers flexible interfaces with datasets and components of a wide range of CTR prediction models. Moreover, we construct a comprehensive system of evaluation protocols encompassing real-world and synthetic datasets, a taxonomy of metrics, standardized procedures and experimental guidelines for calibrating the performance of CTR prediction models. Furthermore, we implement the proposed benchmark platform and conduct a comparative study to evaluate a wide range of state-of-the-art models from traditional multivariate statistical to modern large language model (LLM)-based approaches on three public datasets and two synthetic datasets. Experimental results reveal that, (1) high-order models largely outperform low-order models, though such advantage varies in terms of metrics and on different datasets; (2) LLM-based models demonstrate a remarkable data efficiency, i.e., achieving the comparable performance to other models while using only 2% of the training data; (3) the performance of CTR prediction models has achieved significant improvements from 2015 to 2016, then reached a stage with slow progress, which is consistent across various datasets. This benchmark is expected to facilitate model development and evaluation and enhance practitioners' understanding of the underlying mechanisms of models in the area of CTR prediction. Code is available at https://github.com/NuriaNinja/Bench-CTR.

Related papers

• STEP: Structured Training and Evaluation Platform for benchmarking trajectory prediction models [7.927039780654076]
  We introduce STEP -- a new benchmarking framework that addresses limitations by providing a unified interface for multiple datasets.<n>We demonstrate the capabilities of STEP in a number of experiments which reveal 1) the limitations of widely-used testing procedures, 2) the importance of joint modeling of agents for better predictions of interactions, and 3) the vulnerability of current state-of-the-art models against both distribution shifts and targeted attacks by adversarial agents.
  arXiv  Detail & Related papers  (2025-09-18T09:56:16Z)
• Generative Click-through Rate Prediction with Applications to Search Advertising [6.555660969349762]
  We introduce a novel model that leverages generative models to enhance the precision of CTR predictions in discriminative models.<n>Our method's efficacy is substantiated through extensive experiments on a new dataset.<n>The model is deployed on one of the world's largest e-commerce platforms.
  arXiv  Detail & Related papers  (2025-07-15T12:21:30Z)
• Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis [36.689210473887904]
  We introduce a benchmarking framework for evaluating cross-dataset prediction generalization in deep learning (DL) and machine learning (ML) models.<n>We quantify both absolute performance (e.g., predictive accuracy across datasets) and relative performance (e.g., performance drop compared to within-dataset results)<n>Our results reveal substantial performance drops when models are tested on unseen datasets, underscoring the importance of rigorous generalization assessments.
  arXiv  Detail & Related papers  (2025-03-18T15:40:18Z)
• Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy [104.48511402784763]
  Performance Law for SR models aims to theoretically investigate and model the relationship between model performance and data quality.<n>We propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics.
  arXiv  Detail & Related papers  (2024-11-30T10:56:30Z)
• A Collaborative Ensemble Framework for CTR Prediction [73.59868761656317]
  We propose a novel framework, Collaborative Ensemble Training Network (CETNet), to leverage multiple distinct models. Unlike naive model scaling, our approach emphasizes diversity and collaboration through collaborative learning. We validate our framework on three public datasets and a large-scale industrial dataset from Meta.
  arXiv  Detail & Related papers  (2024-11-20T20:38:56Z)
• PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis [8.785345412061792]
  We introduce a comprehensive framework for modeling single cell transcriptomic responses to perturbations.<n>Our approach includes a modular and user-friendly model development and evaluation platform.<n>We highlight the limitations of widely used models, such as mode collapse.
  arXiv  Detail & Related papers  (2024-08-20T07:40:20Z)
• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance [68.18779562801762]
  multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance. Our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.
  arXiv  Detail & Related papers  (2024-04-04T17:58:02Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction [49.510163437116645]
  Click-through rate (CTR) prediction plays as a core function module in personalized online services. Traditional ID-based models for CTR prediction take as inputs the one-hot encoded ID features of tabular modality. Pretrained Language Models(PLMs) has given rise to another paradigm, which takes as inputs the sentences of textual modality. We propose to conduct Fine-grained feature-level ALignment between ID-based Models and Pretrained Language Models(FLIP) for CTR prediction.
  arXiv  Detail & Related papers  (2023-10-30T11:25:03Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.