Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs

• URL: http://arxiv.org/abs/2406.06768v1
• Date: Mon, 10 Jun 2024 20:06:53 GMT
• Title: Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs
• Abstract summary: We study the design and analysis of switchback experiments conducted on a single aggregate unit. We show that the estimation error depends on four factors: carryover effects, periodicity, serially correlated outcomes, and impacts from simultaneous experiments.
• Score: 1.9376226959814953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the design and analysis of switchback experiments conducted on a single aggregate unit. The design problem is to partition the continuous time space into intervals and switch treatments between intervals, in order to minimize the estimation error of the treatment effect. We show that the estimation error depends on four factors: carryover effects, periodicity, serially correlated outcomes, and impacts from simultaneous experiments. We derive a rigorous bias-variance decomposition and show the tradeoffs of the estimation error from these factors. The decomposition provides three new insights in choosing a design: First, balancing the periodicity between treated and control intervals reduces the variance; second, switching less frequently reduces the bias from carryover effects while increasing the variance from correlated outcomes, and vice versa; third, randomizing interval start and end points reduces both bias and variance from simultaneous experiments. Combining these insights, we propose a new empirical Bayes design approach. This approach uses prior data and experiments for designing future experiments. We illustrate this approach using real data from a ride-sharing platform, yielding a design that reduces MSE by 33% compared to the status quo design used on the platform.

Related papers

• Across-Design Uncertainty in Short Pricing Panels: Inference and Identification [0.0]
  We evaluate the inferential consequences of sparsity in a synthetic data-generating process.<n>Across-design component accounts for 97.6% of estimation error variance for a gradient-boosted specification.
  arXiv  Detail & Related papers  (2026-08-21T17:40:31Z)
• Causal Graph Learning via Distributional Invariance of Cause-Effect Relationship [54.575090553659074]
  We develop an algorithm that efficiently uncovers causal relationships with quadratic complexity in the number of observational variables.<n>Our experiments on a varied benchmark of large-scale datasets show superior or equivalent performance compared to existing works.
  arXiv  Detail & Related papers  (2026-02-03T10:26:16Z)
• Cross-Balancing for Data-Informed Design and Efficient Analysis of Observational Studies [7.899668963928307]
  Cross-balancing is a method that uses sample splitting to separate error in feature construction from the error in weight estimation.<n>We show that careful use of outcome information can substantially improve both estimation and inference while maintaining interpretability.
  arXiv  Detail & Related papers  (2025-11-19T21:56:22Z)
• Variance reduction combining pre-experiment and in-experiment data [0.0]
  Online controlled experiments (A/B testing) are essential in data-driven decision-making for many companies. Existing methods like CUPED and CUPAC use pre-experiment data to reduce variance, but their effectiveness depends on the correlation between the pre-experiment data and the outcome. We introduce a novel method that combines both pre-experiment and in-experiment data to achieve greater variance reduction than CUPED and CUPAC.
  arXiv  Detail & Related papers  (2024-10-11T17:45:29Z)
• Setting the duration of online A/B experiments [0.0]
  This paper provides an analytical formula for the width of a CI based on a ratio treatment effect estimator. We show our formula closely explains CI widths on real A/B experiments at YouTube.
  arXiv  Detail & Related papers  (2024-08-05T20:55:14Z)
• STATE: A Robust ATE Estimator of Heavy-Tailed Metrics for Variance Reduction in Online Controlled Experiments [22.32661807469984]
  We develop a novel framework that integrates the Student's t-distribution with machine learning tools to fit heavy-tailed metrics. By adopting a variational EM method to optimize the loglikehood function, we can infer a robust solution that greatly eliminates the negative impact of outliers. Both simulations on synthetic data and long-term empirical results on Meituan experiment platform demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-07-23T09:35:59Z)
• Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation [53.27596811146316]
  Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts. We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
  arXiv  Detail & Related papers  (2024-01-17T07:58:18Z)
• TCFimt: Temporal Counterfactual Forecasting from Individual Multiple Treatment Perspective [50.675845725806724]
  We propose a comprehensive framework of temporal counterfactual forecasting from an individual multiple treatment perspective (TCFimt) TCFimt constructs adversarial tasks in a seq2seq framework to alleviate selection and time-varying bias and designs a contrastive learning-based block to decouple a mixed treatment effect into separated main treatment effects and causal interactions. The proposed method shows satisfactory performance in predicting future outcomes with specific treatments and in choosing optimal treatment type and timing than state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-12-17T15:01:05Z)
• Averaging Spatio-temporal Signals using Optimal Transport and Soft Alignments [110.79706180350507]
  We show that our proposed loss can be used to define temporal-temporal baryechecenters as Fr'teche means duality. Experiments on handwritten letters and brain imaging data confirm our theoretical findings.
  arXiv  Detail & Related papers  (2022-03-11T09:46:22Z)
• Synthetically Controlled Bandits [2.8292841621378844]
  This paper presents a new dynamic approach to experiment design in settings where, due to interference or other concerns, experimental units are coarse. Our new design, dubbed Synthetically Controlled Thompson Sampling (SCTS), minimizes the regret associated with experimentation at no practically meaningful loss to inferential ability.
  arXiv  Detail & Related papers  (2022-02-14T22:58:13Z)
• Combining Experimental and Observational Data for Identification and Estimation of Long-Term Causal Effects [12.200097942625376]
  We study identifying and estimating the causal effect of a treatment variable on a long-term outcome using data from an observational and an experimental domain.<n>We propose three approaches for data fusion for the purpose of identifying and estimating the causal effect.
  arXiv  Detail & Related papers  (2022-01-26T04:21:14Z)
• What can the millions of random treatments in nonexperimental data reveal about causes? [0.0]
  The article introduces one such model and a Bayesian approach to combine the $O(n2)$ pairwise observations typically available in nonexperimnetal data. We demonstrate that the proposed approach recovers causal effects in common NSW samples, as well as in arbitrary subpopulations and an order-of-magnitude larger supersample.
  arXiv  Detail & Related papers  (2021-05-03T20:13:34Z)
• Efficient Causal Inference from Combined Observational and Interventional Data through Causal Reductions [68.6505592770171]
  Unobserved confounding is one of the main challenges when estimating causal effects. We propose a novel causal reduction method that replaces an arbitrary number of possibly high-dimensional latent confounders. We propose a learning algorithm to estimate the parameterized reduced model jointly from observational and interventional data.
  arXiv  Detail & Related papers  (2021-03-08T14:29:07Z)
• Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates [67.19481956584465]
  It has been experimentally observed that the efficiency of distributed training with saturation (SGD) depends decisively on the batch size and -- in implementations -- on the staleness. We show that our results are tight and illustrate key findings in numerical experiments.
  arXiv  Detail & Related papers  (2021-03-03T12:08:23Z)
• Almost-Matching-Exactly for Treatment Effect Estimation under Network Interference [73.23326654892963]
  We propose a matching method that recovers direct treatment effects from randomized experiments where units are connected in an observed network. Our method matches units almost exactly on counts of unique subgraphs within their neighborhood graphs.
  arXiv  Detail & Related papers  (2020-03-02T15:21:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.