Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED

• URL: http://arxiv.org/abs/2606.18750v1
• Date: Wed, 17 Jun 2026 06:53:03 GMT
• Title: Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED
• Abstract summary: Controlled-experiment Using Pre-Experiment Data (CUPED) is a technique that achieves substantial variance reduction while preserving the unbiasedness of estimating the average treatment effect.<n>This paper systematically addresses five frequently encountered yet overlooked questions regarding the application of CUPED.
• Score: 3.745887419579526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A/B testing has become the gold standard for data-driven decision-making in large-scale online experimentation, providing critical guidance for feature launch, pricing optimization, and user experience enhancement. To maximize statistical sensitivity, many technology companies routinely employ Controlled-experiment Using Pre-Experiment Data (CUPED), a technique that achieves substantial variance reduction while preserving the unbiasedness of estimating the average treatment effect. Despite its widespread adoption, several critical methodological and practical nuances of CUPED remain underexplored. This paper systematically addresses five frequently encountered yet overlooked questions regarding the application of CUPED. First, we provide a comparative analysis of various post-CUPED estimators to identify the optimal adjustment specification. Second, we evaluate the validity of regression-based adjustments and delineate robust variance estimation methods tailored for such frameworks. Finally, we extend our investigation to complex but common scenarios, including multi-arm experiments and two-stage sampling designs. Our findings reveal that in these settings, naive reliance on standard variance estimators can lead to severely misleading inferences. By offering rigorous theoretical insights and extensive experimental validation, this work deepens the conceptual understanding of CUPED. Notably, the recommended methodologies have been successfully deployed and integrated into ByteDance's experimentation platform.

Related papers

• Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap [8.291484471359633]
  We identify a critical inefficiency in the standard A/B-testing protocol.<n>When a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect.<n>We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation.
  arXiv  Detail & Related papers  (2026-07-16T06:08:52Z)
• Unifying On- and Off-Policy Variance Reduction Methods [8.291484471359633]
  We prove that the standard online Difference-in-Means estimator is mathematically identical to an off-policy Inverse Propensity Scoring estimator.<n> Extending this unification, we demonstrate that widespread regression adjustment methods are structurally equivalent to Doubly Robust estimation.
  arXiv  Detail & Related papers  (2026-03-09T13:32:39Z)
• CausalCompass: Evaluating the Robustness of Time-Series Causal Discovery in Misspecified Scenarios [17.11442807888366]
  Causal is a benchmark suite designed to assess the robustness of time-series causal discovery methods under violations of modeling assumptions.<n>We conduct extensive benchmarking of representative TSCD algorithms across eight assumption-violation scenarios.<n>The methods exhibiting superior overall performance across diverse scenarios are almost deep learning-based approaches.
  arXiv  Detail & Related papers  (2026-02-08T11:27:06Z)
• Practical Improvements of A/B Testing with Off-Policy Estimation [51.25970890274447]
  We introduce a family of unbiased off-policy estimators that achieves lower variance than the standard approach.<n>Our theoretical analysis and experimental results validate the effectiveness and practicality of the proposed method.
  arXiv  Detail & Related papers  (2025-06-12T13:11:01Z)
• Post Launch Evaluation of Policies in a High-Dimensional Setting [4.710921988115686]
  A/B tests, also known as randomized controlled experiments (RCTs), are the gold standard for evaluating the impact of new policies, products, or decisions.<n>This paper explores practical considerations in applying methodologies inspired by "synthetic control"<n>Synthetic control methods leverage data from unaffected units to estimate counterfactual outcomes for treated units.
  arXiv  Detail & Related papers  (2024-12-30T19:35:29Z)
• Adaptive Experimentation When You Can't Experiment [55.86593195947978]
  This paper introduces the emphconfounded pure exploration transductive linear bandit (textttCPET-LB) problem. Online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment.
  arXiv  Detail & Related papers  (2024-06-15T20:54:48Z)
• Adapting Prediction Sets to Distribution Shifts Without Labels [16.478151550456804]
  We focus on a standard set-valued prediction framework called conformal prediction (CP)<n>This paper studies how to improve its practical performance using only unlabeled data from the shifted test domain.<n>We show that our methods provide consistent improvement over existing baselines and nearly match the performance of fully supervised methods.
  arXiv  Detail & Related papers  (2024-06-03T15:16:02Z)
• Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
  We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
  arXiv  Detail & Related papers  (2024-05-09T12:52:22Z)
• Adaptive Instrument Design for Indirect Experiments [48.815194906471405]
  Unlike RCTs, indirect experiments estimate treatment effects by leveragingconditional instrumental variables. In this paper we take the initial steps towards enhancing sample efficiency for indirect experiments by adaptively designing a data collection policy. Our main contribution is a practical computational procedure that utilizes influence functions to search for an optimal data collection policy.
  arXiv  Detail & Related papers  (2023-12-05T02:38:04Z)
• Rapid and Scalable Bayesian AB Testing [0.0]
  We propose a solution that applies hierarchical Bayesian estimation to address limitations of current AB testing methodology. We increase statistical power by exploiting correlations between factors, enabling sequential testing and progressive early stopping. We also demonstrate how this methodology can be extended to enable the extraction of composite global learnings from past AB tests.
  arXiv  Detail & Related papers  (2023-07-27T05:08:49Z)
• A Double Machine Learning Approach to Combining Experimental and Observational Data [58.05402364136958]
  We propose a double machine learning approach to combine experimental and observational studies.<n>Our framework proposes a falsification test for external validity and ignorability under milder assumptions.
  arXiv  Detail & Related papers  (2023-07-04T02:53:11Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation [54.72195809248172]
  We present a new estimator leveraging our proposed novel concept, that involves retrospective reshuffling of participants across experimental arms at the end of an RCT. We prove theoretically that such an estimator is more accurate than common estimators based on sample means.
  arXiv  Detail & Related papers  (2023-02-06T05:17:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.