Setting the duration of online A/B experiments

• URL: http://arxiv.org/abs/2408.02830v1
• Date: Mon, 5 Aug 2024 20:55:14 GMT
• Title: Setting the duration of online A/B experiments
• Abstract summary: This paper provides an analytical formula for the width of a CI based on a ratio treatment effect estimator. We show our formula closely explains CI widths on real A/B experiments at YouTube.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In designing an online A/B experiment, it is crucial to select a sample size and duration that ensure the resulting confidence interval (CI) for the treatment effect is the right width to detect an effect of meaningful magnitude with sufficient statistical power without wasting resources. While the relationship between sample size and CI width is well understood, the effect of experiment duration on CI width remains less clear. This paper provides an analytical formula for the width of a CI based on a ratio treatment effect estimator as a function of both sample size (N) and duration (T). The formula is derived from a mixed effects model with two variance components. One component, referred to as the temporal variance, persists over time for experiments where the same users are kept in the same experiment arm across different days. The remaining error variance component, by contrast, decays to zero as T gets large. The formula we derive introduces a key parameter that we call the user-specific temporal correlation (UTC), which quantifies the relative sizes of the two variance components and can be estimated from historical experiments. Higher UTC indicates a slower decay in CI width over time. On the other hand, when the UTC is 0 -- as for experiments where users shuffle in and out of the experiment across days -- the CI width decays at the standard parametric 1/T rate. We also study how access to pre-period data for the users in the experiment affects the CI width decay. We show our formula closely explains CI widths on real A/B experiments at YouTube.

Related papers

• What Time Is It? How Data Geometry Makes Time Conditioning Optional for Flow Matching [30.271879683388033]
  Flow matching models can be trained without explicit time conditioning, challenging the standard view that the time is needed to disambiguate velocity targets.<n>Decomposing the time-blind flow matching loss, we identify two sources of irreducible error.<n>We show that the geometry of high-dimensional data makes time identifiable directly from noisy observations.
  arXiv  Detail & Related papers  (2026-05-08T18:00:36Z)
• A theoretical framework for self-supervised contrastive learning for continuous dependent data [86.50780641055258]
  Self-supervised learning (SSL) has emerged as a powerful approach to learning representations, particularly in the field of computer vision.<n>We propose a novel theoretical framework for contrastive SSL tailored to emphsemantic independence between samples.<n>Specifically, we outperform TS2Vec on the standard UEA and UCR benchmarks, with accuracy improvements of $4.17$% and $2.08$%, respectively.
  arXiv  Detail & Related papers  (2025-06-11T14:23:47Z)
• A Sample Efficient Conditional Independence Test in the Presence of Discretization [54.047334792855345]
  Conditional Independence (CI) tests directly to discretized data can lead to incorrect conclusions.<n>Recent advancements have sought to infer the correct CI relationship between the latent variables through binarizing observed data.<n>Motivated by this, this paper introduces a sample-efficient CI test that does not rely on the binarization process.
  arXiv  Detail & Related papers  (2025-06-10T12:41:26Z)
• Variance reduction combining pre-experiment and in-experiment data [0.0]
  Online controlled experiments (A/B testing) are essential in data-driven decision-making for many companies. Existing methods like CUPED and CUPAC use pre-experiment data to reduce variance, but their effectiveness depends on the correlation between the pre-experiment data and the outcome. We introduce a novel method that combines both pre-experiment and in-experiment data to achieve greater variance reduction than CUPED and CUPAC.
  arXiv  Detail & Related papers  (2024-10-11T17:45:29Z)
• Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs [1.9376226959814953]
  We study the design and analysis of switchback experiments conducted on a single aggregate unit. We show that the estimation error depends on four factors: carryover effects, periodicity, serially correlated outcomes, and impacts from simultaneous experiments.
  arXiv  Detail & Related papers  (2024-06-10T20:06:53Z)
• Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score [62.54911162109439]
  Adversarial detection aims to determine whether a given sample is an adversarial one based on the discrepancy between natural and adversarial distributions. We propose a new statistic called expected perturbation score (EPS), which is essentially the expected score of a sample after various perturbations. We develop EPS-based maximum mean discrepancy (MMD) as a metric to measure the discrepancy between the test sample and natural samples.
  arXiv  Detail & Related papers  (2023-05-25T13:14:58Z)
• Fair Effect Attribution in Parallel Online Experiments [57.13281584606437]
  A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services. It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly. Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes.
  arXiv  Detail & Related papers  (2022-10-15T17:15:51Z)
• Full time-dependent counting statistics of highly entangled biphoton states [0.0]
  This paper presents an approach providing full time-dependent counting statistics in terms of efficiently computable formulas. General spatial modes are taken into account to describe free space and fiber propagation. The approach is easily applicable to a modular array of arbitrary optical components and external influences.
  arXiv  Detail & Related papers  (2022-09-08T12:45:18Z)
• H\"older Bounds for Sensitivity Analysis in Causal Reasoning [66.00472443147781]
  We derive a set of bounds on the confounding bias |E[Y|T=t]-E[Y|do(T=t)]| based on the degree of unmeasured confounding. These bounds are tight either when U is independent of T or when U is independent of Y given T.
  arXiv  Detail & Related papers  (2021-07-09T20:26:36Z)
• Bayesian Kernelised Test of (In)dependence with Mixed-type Variables [1.2691047660244332]
  A fundamental task in AI is to assess (in)dependence between mixed-type variables (text, image, sound) We propose a Bayesian kernelised correlation test of (in)dependence using a Dirichlet process model. We show the properties of the approach, as well as algorithms for fast computation with it.
  arXiv  Detail & Related papers  (2021-05-09T19:21:43Z)
• Multi-Source Causal Inference Using Control Variates [81.57072928775509]
  We propose a general algorithm to estimate causal effects from emphmultiple data sources. We show theoretically that this reduces the variance of the ATE estimate. We apply this framework to inference from observational data under an outcome selection bias.
  arXiv  Detail & Related papers  (2021-03-30T21:20:51Z)
• Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates [67.19481956584465]
  It has been experimentally observed that the efficiency of distributed training with saturation (SGD) depends decisively on the batch size and -- in implementations -- on the staleness. We show that our results are tight and illustrate key findings in numerical experiments.
  arXiv  Detail & Related papers  (2021-03-03T12:08:23Z)
• Correct block-design experiments mitigate temporal correlation bias in EEG classification [68.85562949901077]
  We show that the main claim in [1] is drastically overstated and their other analyses are seriously flawed by wrong methodological choices. We investigate the influence of EEG temporal correlation on classification accuracy by testing the same models in two additional experimental settings.
  arXiv  Detail & Related papers  (2020-11-25T22:25:21Z)
• A Case-Study on the Impact of Dynamic Time Warping in Time Series Regression [2.639737913330821]
  We show that Dynamic Time Warping (DTW) is effective in improving accuracy on a regression task when only a single wavelength is considered. When combined with k-Nearest Neighbour, DTW has the added advantage that it can reveal similarities and differences between samples at the level of the time-series. However, in the problem, we consider here data is available across a spectrum of wavelengths.
  arXiv  Detail & Related papers  (2020-10-11T15:21:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.