Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

• URL: http://arxiv.org/abs/2607.14604v1
• Date: Thu, 16 Jul 2026 06:08:52 GMT
• Title: Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap
• Abstract summary: We identify a critical inefficiency in the standard A/B-testing protocol.<n>When a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect.<n>We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation.
• Score: 8.291484471359633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online controlled experiments are the gold standard for hypothesis testing in online platforms. Notwithstanding their ubiquity, they are notoriously expensive to run, and issues of variance hamper statistical power in assessing treatment effects. While standard variance reduction techniques leverage model-based control variates to reduce outcome noise, they remain agnostic to potential structural relationships between competing policies. In this work, we identify a critical inefficiency in the standard A/B-testing protocol: when a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect -- unnecessarily inflating confidence intervals. We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation. The key insight is to frame the randomised treatment assignment mechanism as a meta-policy, and leverage $Δ$-Off-Policy Estimation methods to obtain unbiased estimates for average treatment effects. We prove analytically that our approach recovers standard A/B-testing practices in the general case, but that its variance scales with the divergence between policies rather than raw outcome variance. Hence, we dominate the standard Difference-in-Means estimator whenever policies have common support, and the improvement is strict whenever the overlap region contributes non-zero residual variance. Empirical results corroborate these theoretical insights -- holding promise for significant impact on the real-world evaluation of recommender systems, information retrieval pipelines, and large language model interfaces.

Related papers

• Unifying On- and Off-Policy Variance Reduction Methods [8.291484471359633]
  We prove that the standard online Difference-in-Means estimator is mathematically identical to an off-policy Inverse Propensity Scoring estimator.<n> Extending this unification, we demonstrate that widespread regression adjustment methods are structurally equivalent to Doubly Robust estimation.
  arXiv  Detail & Related papers  (2026-03-09T13:32:39Z)
• Beyond Normality: Reliable A/B Testing with Non-Gaussian Data [15.568830806973407]
  We quantify how skewed, long tailed data and unequal allocation distort error rates and derive explicit formulas for the minimum sample size required for the $t$-test to remain valid.<n>We find that many online feedback metrics require hundreds of millions samples to ensure reliable A/B testing.
  arXiv  Detail & Related papers  (2025-10-26T14:44:19Z)
• Practical Improvements of A/B Testing with Off-Policy Estimation [51.25970890274447]
  We introduce a family of unbiased off-policy estimators that achieves lower variance than the standard approach.<n>Our theoretical analysis and experimental results validate the effectiveness and practicality of the proposed method.
  arXiv  Detail & Related papers  (2025-06-12T13:11:01Z)
• Prediction-Powered Causal Inferences [59.98498488132307]
  We focus on Prediction-Powered Causal Inferences (PPCI)<n>We first show that conditional calibration guarantees valid PPCI at population level.<n>We then introduce a sufficient representation constraint transferring validity across experiments.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Post Launch Evaluation of Policies in a High-Dimensional Setting [4.710921988115686]
  A/B tests, also known as randomized controlled experiments (RCTs), are the gold standard for evaluating the impact of new policies, products, or decisions.<n>This paper explores practical considerations in applying methodologies inspired by "synthetic control"<n>Synthetic control methods leverage data from unaffected units to estimate counterfactual outcomes for treated units.
  arXiv  Detail & Related papers  (2024-12-30T19:35:29Z)
• Improving the Estimation of Lifetime Effects in A/B Testing via Treatment Locality [16.36651676133996]
  We develop optimal inference techniques for general A/B testing in Markov Decision Processes.<n>We propose methods to harness the localized structure by sharing information on the non-targeted states.<n>We show that all such estimators can benefit from variance reduction through information sharing without increasing their bias.
  arXiv  Detail & Related papers  (2024-07-29T00:41:11Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Conformal Off-Policy Evaluation in Markov Decision Processes [53.786439742572995]
  Reinforcement Learning aims at identifying and evaluating efficient control policies from data. Most methods for this learning task, referred to as Off-Policy Evaluation (OPE), do not come with accuracy and certainty guarantees. We present a novel OPE method based on Conformal Prediction that outputs an interval containing the true reward of the target policy with a prescribed level of certainty.
  arXiv  Detail & Related papers  (2023-04-05T16:45:11Z)
• Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation [54.72195809248172]
  We present a new estimator leveraging our proposed novel concept, that involves retrospective reshuffling of participants across experimental arms at the end of an RCT. We prove theoretically that such an estimator is more accurate than common estimators based on sample means.
  arXiv  Detail & Related papers  (2023-02-06T05:17:22Z)
• Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Bandits [5.144809478361604]
  We improve the doubly robust (DR) estimator by adaptively weighting observations to control its variance. We provide empirical evidence for our estimator's improved accuracy and inferential properties relative to existing alternatives.
  arXiv  Detail & Related papers  (2021-06-03T17:54:44Z)
• Post-Contextual-Bandit Inference [57.88785630755165]
  Contextual bandit algorithms are increasingly replacing non-adaptive A/B tests in e-commerce, healthcare, and policymaking. They can both improve outcomes for study participants and increase the chance of identifying good or even best policies. To support credible inference on novel interventions at the end of the study, we still want to construct valid confidence intervals on average treatment effects, subgroup effects, or value of new policies.
  arXiv  Detail & Related papers  (2021-06-01T12:01:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.