ML-assisted Randomization Tests for Detecting Treatment Effects in A/B Experiments

• URL: http://arxiv.org/abs/2501.07722v1
• Date: Mon, 13 Jan 2025 22:14:58 GMT
• Title: ML-assisted Randomization Tests for Detecting Treatment Effects in A/B Experiments
• Authors: Wenxuan Guo, JungHo Lee, Panos Toulis,
• Abstract summary: In this paper, we construct randomization tests for complex treatment effects. A key feature of our approach is the use of flexible machine learning (ML) models. This approach combines the predictive power of modern ML tools with the finite-sample validity of randomization procedures.
• Score: 3.79377147545355
• License:
• Abstract: Experimentation is widely utilized for causal inference and data-driven decision-making across disciplines. In an A/B experiment, for example, an online business randomizes two different treatments (e.g., website designs) to their customers and then aims to infer which treatment is better. In this paper, we construct randomization tests for complex treatment effects, including heterogeneity and interference. A key feature of our approach is the use of flexible machine learning (ML) models, where the test statistic is defined as the difference between the cross-validation errors from two ML models, one including the treatment variable and the other without it. This approach combines the predictive power of modern ML tools with the finite-sample validity of randomization procedures, enabling a robust and efficient way to detect complex treatment effects in experimental settings. We demonstrate this combined benefit both theoretically and empirically through applied examples.

Related papers

• A Simple Model to Estimate Sharing Effects in Social Networks [3.988614978933934]
  We propose a simple Markov Decision Process (MDP)-based model describing user sharing behaviour in social networks. We derive an unbiased estimator for treatment effects under this model, and demonstrate through reproducible synthetic experiments that it outperforms existing methods by a significant margin.
  arXiv  Detail & Related papers  (2024-09-16T13:32:36Z)
• Improving Bias Correction Standards by Quantifying its Effects on Treatment Outcomes [54.18828236350544]
  Propensity score matching (PSM) addresses selection biases by selecting comparable populations for analysis. Different matching methods can produce significantly different Average Treatment Effects (ATE) for the same task, even when meeting all validation criteria. To address this issue, we introduce a novel metric, A2A, to reduce the number of valid matches.
  arXiv  Detail & Related papers  (2024-07-20T12:42:24Z)
• Adaptive Experimentation When You Can't Experiment [55.86593195947978]
  This paper introduces the emphconfounded pure exploration transductive linear bandit (textttCPET-LB) problem. Online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment.
  arXiv  Detail & Related papers  (2024-06-15T20:54:48Z)
• CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective [48.99488315273868]
  We present a contrastive knowledge distillation approach, which can be formulated as a sample-wise alignment problem with intra- and inter-sample constraints. Our method minimizes logit differences within the same sample by considering their numerical values. We conduct comprehensive experiments on three datasets including CIFAR-100, ImageNet-1K, and MS COCO.
  arXiv  Detail & Related papers  (2024-04-22T11:52:40Z)
• Machine Learning Assisted Adjustment Boosts Efficiency of Exact Inference in Randomized Controlled Trials [12.682443719767763]
  We show the proposed method can robustly control the type I error and can boost the statistical efficiency for a randomized controlled trial (RCT) Its application may remarkably reduce the required sample size and cost of RCTs, such as phase III clinical trials.
  arXiv  Detail & Related papers  (2024-03-05T15:48:07Z)
• Individualized Multi-Treatment Response Curves Estimation using RBF-net with Shared Neurons [1.1119247609126184]
  Our non-parametric modeling of the response curves relies on radial basis function (RBF)-nets with shared hidden neurons. Applying our proposed method to MIMIC data, we obtain several interesting findings related to the impact of different treatment strategies on the length of ICU stay and 12-hour SOFA score for sepsis patients who are home-discharged.
  arXiv  Detail & Related papers  (2024-01-29T21:13:01Z)
• Adaptive Instrument Design for Indirect Experiments [48.815194906471405]
  Unlike RCTs, indirect experiments estimate treatment effects by leveragingconditional instrumental variables. In this paper we take the initial steps towards enhancing sample efficiency for indirect experiments by adaptively designing a data collection policy. Our main contribution is a practical computational procedure that utilizes influence functions to search for an optimal data collection policy.
  arXiv  Detail & Related papers  (2023-12-05T02:38:04Z)
• The Blessings of Multiple Treatments and Outcomes in Treatment Effect Estimation [53.81860494566915]
  Existing studies leveraged proxy variables or multiple treatments to adjust for confounding bias. In many real-world scenarios, there is greater interest in studying the effects on multiple outcomes. We show that parallel studies of multiple outcomes involved in this setting can assist each other in causal identification.
  arXiv  Detail & Related papers  (2023-09-29T14:33:48Z)
• Comparison of Methods that Combine Multiple Randomized Trials to Estimate Heterogeneous Treatment Effects [0.1398098625978622]
  Leveraging multiple randomized controlled trials allows for the combination of datasets with unconfounded treatment assignment. This paper discusses several non-parametric approaches for estimating heterogeneous treatment effects using data from multiple trials.
  arXiv  Detail & Related papers  (2023-03-28T20:43:00Z)
• Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments [0.9208007322096533]
  We develop a general approach to statistical inference for heterogeneous treatment effects discovered by a generic ML algorithm. We show how to estimate the average treatment effect within each of these groups, and construct a valid confidence interval.
  arXiv  Detail & Related papers  (2022-03-28T05:43:46Z)
• Localized Debiased Machine Learning: Efficient Inference on Quantile Treatment Effects and Beyond [69.83813153444115]
  We consider an efficient estimating equation for the (local) quantile treatment effect ((L)QTE) in causal inference. Debiased machine learning (DML) is a data-splitting approach to estimating high-dimensional nuisances. We propose localized debiased machine learning (LDML), which avoids this burdensome step.
  arXiv  Detail & Related papers  (2019-12-30T14:42:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.