Comparison Lift: Bandit-based Experimentation System for Online Advertising

• URL: http://arxiv.org/abs/2009.07899v1
• Date: Wed, 16 Sep 2020 19:08:44 GMT
• Title: Comparison Lift: Bandit-based Experimentation System for Online Advertising
• Abstract summary: Comparison Lift is an experimentation-as-a-service (E) application for testing online advertising audiences and creatives at JD.com. Unlike many other E tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design.
• Score: 10.499720681099452
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Comparison Lift is an experimentation-as-a-service (EaaS) application for testing online advertising audiences and creatives at JD.com. Unlike many other EaaS tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. The advantages of the bandit-based approach are two-fold. First, it aligns the randomization induced in the test with the advertiser's goals from testing. Second, by adapting experimental design to information acquired during the test, it reduces substantially the cost of experimentation to the advertiser. Since launch in May 2019, Comparison Lift has been utilized in over 1,500 experiments. We estimate that utilization of the product has helped increase click-through rates of participating advertising campaigns by 46% on average. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design. Both suggest significant value generation and cost savings to advertisers from the product.

Related papers

• Adaptive Ad Load Design for Sponsored Search Markets: Evidence, Theory, and Deployment [0.0]
  Ad-load design is a central supply-side decision in sponsored search.<n>We study this trade-off using a large-scale randomized field experiment on an Android app store.<n>We deploy a novel adaptive algorithm -- exploration-augmented Locally Adaptive Ad Load (e-LAAL)
  arXiv  Detail & Related papers  (2026-07-15T23:12:26Z)
• A Large Scale Heterogeneous Treatment Effect Estimation Framework and Its Applications of Users' Journey at Snap [5.833463670402037]
  Heterogeneous Treatment Effect (HTE) and Conditional Average Treatment Effect (CATE) models relax the assumption that treatment effects are the same for every user.<n>We present a large scale industrial framework for estimating HTE using experimental data from hundreds of millions of Snapchat users.
  arXiv  Detail & Related papers  (2025-11-25T22:01:58Z)
• Practical Improvements of A/B Testing with Off-Policy Estimation [51.25970890274447]
  We introduce a family of unbiased off-policy estimators that achieves lower variance than the standard approach.<n>Our theoretical analysis and experimental results validate the effectiveness and practicality of the proposed method.
  arXiv  Detail & Related papers  (2025-06-12T13:11:01Z)
• Using Auxiliary Data to Boost Precision in the Analysis of A/B Tests on an Online Educational Platform: New Data and New Results [1.5293427903448025]
  A/B tests allow causal effect estimation without confounding bias and exact statistical inference even in small samples. Recent methodological advances have shown that power and statistical precision can be substantially boosted by coupling design-based causal estimation to machine-learning models of rich log data from historical users who were not in the experiment. We show that the gains can be even larger for estimating subgroup effects, hold even when the remnant is unrepresentative of the A/B test sample, and extend to post-stratification population effects estimators.
  arXiv  Detail & Related papers  (2023-06-09T21:54:36Z)
• Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobao [24.43583745735832]
  We propose to leverage recommendations samples as a free lunch to mitigate sample selection bias for ads CTR model (Rec4Ad) Rec4Ad achieves substantial gains in key business metrics, with a lift of up to +6.6% CTR and +2.9% RPM.
  arXiv  Detail & Related papers  (2023-06-06T09:22:52Z)
• Conversion-Based Dynamic-Creative-Optimization in Native Advertising [2.8461446020965435]
  Dynamic creative optimization (DCO) is a recent Gemini native product that was launched two years ago and is increasingly gaining more attention from advertisers. To accommodate this request, we present a post-auction solution, where DCO ads combinations are favored according to their predicted conversion rate (CVR) An online evaluation of this explore-exploit solution, via online bucket A/B testing, showed a 53.5% CVR lift, when compared to a control bucket serving all combinations uniformly at random.
  arXiv  Detail & Related papers  (2022-11-13T13:13:00Z)
• Privacy Aware Experiments without Cookies [13.488622527004082]
  Two brands want to jointly test alternate web experiences for their customers with an A/B test. Such collaborative tests are today enabled using textitthird-party cookies, where each brand has information on the identity of visitors to another website. With the imminent elimination of third-party cookies, such A/B tests will become untenable. We propose a two-stage experimental design, where the two brands only need to agree on high-level aggregate parameters of the experiment to test the alternate experiences.
  arXiv  Detail & Related papers  (2022-11-03T23:44:01Z)
• ReSmooth: Detecting and Utilizing OOD Samples when Training with Data Augmentation [57.38418881020046]
  Recent DA techniques always meet the need for diversity in augmented training samples. An augmentation strategy that has a high diversity usually introduces out-of-distribution (OOD) augmented samples. We propose ReSmooth, a framework that firstly detects OOD samples in augmented samples and then leverages them.
  arXiv  Detail & Related papers  (2022-05-25T09:29:27Z)
• Adaptive Experimentation with Delayed Binary Feedback [11.778924435036519]
  This paper presents an adaptive experimentation solution tailored for delayed binary feedback objectives. It estimates the real underlying objectives before they materialize and dynamically allocates variants based on the estimates. This solution is currently deployed in the online experimentation platform of JD.com.
  arXiv  Detail & Related papers  (2022-02-02T01:47:10Z)
• SelectAugment: Hierarchical Deterministic Sample Selection for Data Augmentation [72.58308581812149]
  We propose an effective approach, dubbed SelectAugment, to select samples to be augmented in a deterministic and online manner. Specifically, in each batch, we first determine the augmentation ratio, and then decide whether to augment each training sample under this ratio. In this way, the negative effects of the randomness in selecting samples to augment can be effectively alleviated and the effectiveness of DA is improved.
  arXiv  Detail & Related papers  (2021-12-06T08:38:38Z)
• Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees [63.62448343531963]
  We propose a combination of the existing paradigms, sampling responses to be scored by humans intelligently. We observe significant gains in accuracy (19.80% increase on average) and quadratic weighted kappa (QWK) (25.60% on average) with a relatively small human budget.
  arXiv  Detail & Related papers  (2021-11-17T05:00:51Z)
• Do Interruptions Pay Off? Effects of Interruptive Ads on Consumers Willingness to Pay [79.9312329825761]
  We present the results of a study designed to measure the impact of interruptive advertising on consumers willingness to pay for products bearing the advertiser's brand. Our results contribute to the research on the economic impact of advertising, and introduce a method of measuring actual (as opposed to self-reported) willingness to pay in experimental marketing research.
  arXiv  Detail & Related papers  (2020-05-14T09:26:57Z)
• Noisy Adaptive Group Testing using Bayesian Sequential Experimental Design [63.48989885374238]
  When the infection prevalence of a disease is low, Dorfman showed 80 years ago that testing groups of people can prove more efficient than testing people individually. Our goal in this paper is to propose new group testing algorithms that can operate in a noisy setting.
  arXiv  Detail & Related papers  (2020-04-26T23:41:33Z)
• Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework [68.96770035057716]
  A/B testing is a business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. This paper introduces a reinforcement learning framework for carrying A/B testing in online experiments.
  arXiv  Detail & Related papers  (2020-02-05T10:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.