Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling

• URL: http://arxiv.org/abs/2104.14121v1
• Date: Thu, 29 Apr 2021 05:37:34 GMT
• Title: Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling
• Authors: Siyu Gu, Xiang-Rong Sheng, Ying Fan, Guorui Zhou, Xiaoqiang Zhu
• Abstract summary: We propose DElayed FEedback modeling with Real negatives (DEFER) method to address these issues. The ingestion of real negatives ensures the observed feature distribution is equivalent to the actual distribution, thus reducing the bias. DEFER have been deployed in the display advertising system of Alibaba, obtaining over 6.4% improvement on CVR in several scenarios.
• Score: 10.828167195122072
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: One of the difficulties of conversion rate (CVR) prediction is that the conversions can delay and take place long after the clicks. The delayed feedback poses a challenge: fresh data are beneficial to continuous training but may not have complete label information at the time they are ingested into the training pipeline. To balance model freshness and label certainty, previous methods set a short waiting window or even do not wait for the conversion signal. If conversion happens outside the waiting window, this sample will be duplicated and ingested into the training pipeline with a positive label. However, these methods have some issues. First, they assume the observed feature distribution remains the same as the actual distribution. But this assumption does not hold due to the ingestion of duplicated samples. Second, the certainty of the conversion action only comes from the positives. But the positives are scarce as conversions are sparse in commercial systems. These issues induce bias during the modeling of delayed feedback. In this paper, we propose DElayed FEedback modeling with Real negatives (DEFER) method to address these issues. The proposed method ingests real negative samples into the training pipeline. The ingestion of real negatives ensures the observed feature distribution is equivalent to the actual distribution, thus reducing the bias. The ingestion of real negatives also brings more certainty information of the conversion. To correct the distribution shift, DEFER employs importance sampling to weigh the loss function. Experimental results on industrial datasets validate the superiority of DEFER. DEFER have been deployed in the display advertising system of Alibaba, obtaining over 6.0% improvement on CVR in several scenarios. The code and data in this paper are now open-sourced {https://github.com/gusuperstar/defer.git}.

Related papers

• Freshness or Accuracy, Why Not Both? Addressing Delayed Feedback via Dynamic Graph Neural Networks [23.952923773407043]
  Delayed feedback problem is one of the most pressing challenges in predicting conversion rate. We propose Delayed Feedback Modeling by Dynamic Graph Neural Network (DGDFEM) It includes three stages, i.e., preparing a data pipeline, building a dynamic graph, and training a CVR prediction model.
  arXiv  Detail & Related papers  (2023-08-15T23:49:07Z)
• Making Pre-trained Language Models both Task-solvers and Self-calibrators [52.98858650625623]
  Pre-trained language models (PLMs) serve as backbones for various real-world systems. Previous work shows that introducing an extra calibration task can mitigate this issue. We propose a training algorithm LM-TOAST to tackle the challenges.
  arXiv  Detail & Related papers  (2023-07-21T02:51:41Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)
• Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction [14.462884375151045]
  Delayed feedback is crucial for the conversion rate prediction in online advertising. Previous delayed feedback modeling methods balance the trade-off between waiting for accurate labels and consuming fresh feedback. We propose a new method, DElayed Feedback modeling with UnbiaSed Estimation, (DEFUSE), which aim to respectively correct the importance weights of the immediate positive, the fake negative, the real negative, and the delay positive samples.
  arXiv  Detail & Related papers  (2022-02-14T03:31:09Z)
• Agree to Disagree: Diversity through Disagreement for Better Transferability [54.308327969778155]
  We propose D-BAT (Diversity-By-disAgreement Training), which enforces agreement among the models on the training data. We show how D-BAT naturally emerges from the notion of generalized discrepancy.
  arXiv  Detail & Related papers  (2022-02-09T12:03:02Z)
• Tracking the risk of a deployed model and detecting harmful distribution shifts [105.27463615756733]
  In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially. We argue that a sensible method for firing off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate.
  arXiv  Detail & Related papers  (2021-10-12T17:21:41Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)
• Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling [29.77426549280091]
  Conversion rate (CVR) prediction is one of the most critical tasks for digital display advertising. We propose Elapsed-Time Sampling Delayed Feedback Model (ES-DFM), which models the relationship between the observed conversion distribution and the true conversion distribution.
  arXiv  Detail & Related papers  (2020-12-06T12:20:50Z)
• Learning Classifiers under Delayed Feedback with a Time Window Assumption [16.269923100433235]
  We consider training a binary classifier under delayed feedback (emphDF learning) We initially receive negative samples; subsequently, some samples among them change to positive. Owing to the delayed feedback, naive classification of the positive and negative samples returns a biased classifier.
  arXiv  Detail & Related papers  (2020-09-28T06:20:24Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)
• A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback [6.38500614968955]
  In display advertising, predicting the conversion rate is fundamental to estimating the value of displaying the advertisement. There is a relatively long time delay between a click and its resultant conversion. Because of the delayed feedback, some positive instances at the training period are labeled as negative.
  arXiv  Detail & Related papers  (2020-02-06T02:05:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.