Powerful A/B-Testing Metrics and Where to Find Them

• URL: http://arxiv.org/abs/2407.20665v1
• Date: Tue, 30 Jul 2024 08:59:50 GMT
• Title: Powerful A/B-Testing Metrics and Where to Find Them
• Authors: Olivier Jeunen, Shubham Baweja, Neeti Pokharna, Aleksei Ustimenko,
• Abstract summary: A/B-tests are the bread and butter of real-world recommender system evaluation. A North Star metric is used to assess which system variant should be deemed superior. We propose to collect this information and leverage it to quantify type-I, type-II, and type-III errors for the metrics of interest. We present results and insights from building this pipeline at scale for two large-scale short-video platforms: ShareChat and Moj.
• Score: 11.018341970786574
• License:
• Abstract: Online controlled experiments, colloquially known as A/B-tests, are the bread and butter of real-world recommender system evaluation. Typically, end-users are randomly assigned some system variant, and a plethora of metrics are then tracked, collected, and aggregated throughout the experiment. A North Star metric (e.g. long-term growth or revenue) is used to assess which system variant should be deemed superior. As a result, most collected metrics are supporting in nature, and serve to either (i) provide an understanding of how the experiment impacts user experience, or (ii) allow for confident decision-making when the North Star metric moves insignificantly (i.e. a false negative or type-II error). The latter is not straightforward: suppose a treatment variant leads to fewer but longer sessions, with more views but fewer engagements; should this be considered a positive or negative outcome? The question then becomes: how do we assess a supporting metric's utility when it comes to decision-making using A/B-testing? Online platforms typically run dozens of experiments at any given time. This provides a wealth of information about interventions and treatment effects that can be used to evaluate metrics' utility for online evaluation. We propose to collect this information and leverage it to quantify type-I, type-II, and type-III errors for the metrics of interest, alongside a distribution of measurements of their statistical power (e.g. $z$-scores and $p$-values). We present results and insights from building this pipeline at scale for two large-scale short-video platforms: ShareChat and Moj; leveraging hundreds of past experiments to find online metrics with high statistical power.

Related papers

• Comprehensive Equity Index (CEI): Definition and Application to Bias Evaluation in Biometrics [47.762333925222926]
  We present a novel metric to quantify biased behaviors of machine learning models. We focus on and apply it to the operational evaluation of face recognition systems.
  arXiv  Detail & Related papers  (2024-09-03T14:19:38Z)
• Is Reference Necessary in the Evaluation of NLG Systems? When and Where? [58.52957222172377]
  We show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.
  arXiv  Detail & Related papers  (2024-03-21T10:31:11Z)
• Cobra Effect in Reference-Free Image Captioning Metrics [58.438648377314436]
  A proliferation of reference-free methods, leveraging visual-language pre-trained models (VLMs), has emerged. In this paper, we study if there are any deficiencies in reference-free metrics. We employ GPT-4V as an evaluative tool to assess generated sentences and the result reveals that our approach achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2024-02-18T12:36:23Z)
• Learning Metrics that Maximise Power for Accelerated A/B-Tests [13.528097424046823]
  North Star metrics are typically delayed and insensitive. Experiments need to run for a long time, and even then, type-II errors are prevalent. We propose to tackle this by learning metrics from short-term signals.
  arXiv  Detail & Related papers  (2024-02-06T11:31:04Z)
• Variance Reduction in Ratio Metrics for Efficient Online Experiments [12.036747050794135]
  We apply variance reduction techniques to ratio metrics on a large-scale short-video platform: ShareChat. Our results show that we can either improve A/B-test confidence in 77% of cases, or can retain the same level of confidence with 30% fewer data points.
  arXiv  Detail & Related papers  (2024-01-08T18:01:09Z)
• Choosing a Proxy Metric from Past Experiments [54.338884612982405]
  In many randomized experiments, the treatment effect of the long-term metric is often difficult or infeasible to measure. A common alternative is to measure several short-term proxy metrics in the hope they closely track the long-term metric. We introduce a new statistical framework to both define and construct an optimal proxy metric for use in a homogeneous population of randomized experiments.
  arXiv  Detail & Related papers  (2023-09-14T17:43:02Z)
• A Common Misassumption in Online Experiments with Machine Learning Models [1.52292571922932]
  We argue that, because variants typically learn using pooled data, a lack of model interference cannot be guaranteed. We discuss the implications this has for practitioners, and for the research literature.
  arXiv  Detail & Related papers  (2023-04-21T11:36:44Z)
• Choose Your Lenses: Flaws in Gender Bias Evaluation [29.16221451643288]
  We assess the current paradigm of gender bias evaluation and identify several flaws in it. First, we highlight the importance of extrinsic bias metrics that measure how a model's performance on some task is affected by gender. Second, we find that datasets and metrics are often coupled, and discuss how their coupling hinders the ability to obtain reliable conclusions.
  arXiv  Detail & Related papers  (2022-10-20T17:59:55Z)
• Clustering-based Imputation for Dropout Buyers in Large-scale Online Experimentation [4.753069295451989]
  In online experimentation, appropriate metrics (e.g., purchase) provide strong evidence to support hypotheses and enhance the decision-making process. In this work, we introduce the concept of dropout buyers and categorize users with incomplete metric values into two groups: visitors and dropout buyers. For the analysis of incomplete metrics, we propose a clustering-based imputation method using $k$-nearest neighbors.
  arXiv  Detail & Related papers  (2022-09-09T01:05:53Z)
• On Quantitative Evaluations of Counterfactuals [88.42660013773647]
  This paper consolidates work on evaluating visual counterfactual examples through an analysis and experiments. We find that while most metrics behave as intended for sufficiently simple datasets, some fail to tell the difference between good and bad counterfactuals when the complexity increases. We propose two new metrics, the Label Variation Score and the Oracle score, which are both less vulnerable to such tiny changes.
  arXiv  Detail & Related papers  (2021-10-30T05:00:36Z)
• Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework [68.96770035057716]
  A/B testing is a business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. This paper introduces a reinforcement learning framework for carrying A/B testing in online experiments.
  arXiv  Detail & Related papers  (2020-02-05T10:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.