Inference for Batched Adaptive Experiments

• URL: http://arxiv.org/abs/2512.10156v1
• Date: Wed, 10 Dec 2025 23:33:08 GMT
• Title: Inference for Batched Adaptive Experiments
• Authors: Jan Kemper, Davud Rostam-Afschar,
• Abstract summary: This note suggests a BOLS test statistic for inference of treatment effects in adaptive experiments.<n>We provide simulation results comparing rejection rates in the typical case with few treatment periods and few (or many) observations per batch.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The advantages of adaptive experiments have led to their rapid adoption in economics, other fields, as well as among practitioners. However, adaptive experiments pose challenges for causal inference. This note suggests a BOLS (batched ordinary least squares) test statistic for inference of treatment effects in adaptive experiments. The statistic provides a precision-equalizing aggregation of per-period treatment-control differences under heteroskedasticity. The combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals. We provide simulation results comparing rejection rates in the typical case with few treatment periods and few (or many) observations per batch.

Related papers

• Testing Effect Homogeneity and Confounding in High-Dimensional Experimental and Observational Studies [0.552480439325792]
  We propose a framework for testing the homogeneity of conditional average treatment effects (CATEs) across multiple experimental and observational studies.<n>Our approach leverages multiple randomized trials to assess whether treatment effects vary with unobserved heterogeneity that differs across trials.
  arXiv  Detail & Related papers  (2026-02-23T10:52:39Z)
• A Sample Efficient Conditional Independence Test in the Presence of Discretization [54.047334792855345]
  Conditional Independence (CI) tests directly to discretized data can lead to incorrect conclusions.<n>Recent advancements have sought to infer the correct CI relationship between the latent variables through binarizing observed data.<n>Motivated by this, this paper introduces a sample-efficient CI test that does not rely on the binarization process.
  arXiv  Detail & Related papers  (2025-06-10T12:41:26Z)
• Multi-Metric Adaptive Experimental Design under Fixed Budget with Validation [10.5481503979787]
  Standard A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously.<n>This paper proposes a fixed-budget multi-metric AED framework with a two-phase structure: an adaptive exploration phase to identify the best treatment, and a validation phase to verify the treatment's quality and infer statistics.
  arXiv  Detail & Related papers  (2025-06-03T16:41:11Z)
• Prediction-Powered Causal Inferences [59.98498488132307]
  We focus on Prediction-Powered Causal Inferences (PPCI)<n>We first show that conditional calibration guarantees valid PPCI at population level.<n>We then introduce a sufficient representation constraint transferring validity across experiments.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Efficient Randomized Experiments Using Foundation Models [16.80288371969967]
  In this paper, we propose a novel approach that integrates the predictions from multiple foundation models while preserving valid statistical inference.<n>Our estimator offers substantial precision gains, equivalent to a reduction of up to 20% in the sample size needed to match the same precision as the standard estimator based on experimental data alone.
  arXiv  Detail & Related papers  (2025-02-06T17:54:10Z)
• Demistifying Inference after Adaptive Experiments [43.653628046172656]
  Adaptive experiments such as multi-arm bandits adapt the treatment-allocation policy and/or the decision to stop the experiment to the data observed so far. The concentration inequalities and union bounds that generally underlie adaptive experimentation algorithms can yield overly conservative inferences. In this article we aim to explain why, how, and when adaptivity is in fact an issue for inference and, when it is, understand the various ways to fix it.
  arXiv  Detail & Related papers  (2024-05-02T13:39:51Z)
• Adaptive Instrument Design for Indirect Experiments [48.815194906471405]
  Unlike RCTs, indirect experiments estimate treatment effects by leveragingconditional instrumental variables. In this paper we take the initial steps towards enhancing sample efficiency for indirect experiments by adaptively designing a data collection policy. Our main contribution is a practical computational procedure that utilizes influence functions to search for an optimal data collection policy.
  arXiv  Detail & Related papers  (2023-12-05T02:38:04Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Learning sources of variability from high-dimensional observational studies [41.06757602546625]
  Causal inference studies whether the presence of a variable influences an observed outcome. Our work generalizes causal estimands to outcomes with any number of dimensions or any measurable space. We propose a simple technique for adjusting universally consistent conditional independence tests.
  arXiv  Detail & Related papers  (2023-07-26T00:01:16Z)
• DELTA: degradation-free fully test-time adaptation [59.74287982885375]
  We find that two unfavorable defects are concealed in the prevalent adaptation methodologies like test-time batch normalization (BN) and self-learning. First, we reveal that the normalization statistics in test-time BN are completely affected by the currently received test samples, resulting in inaccurate estimates. Second, we show that during test-time adaptation, the parameter update is biased towards some dominant classes.
  arXiv  Detail & Related papers  (2023-01-30T15:54:00Z)
• Counterfactual Inference of the Mean Outcome under a Convergence of Average Logging Probability [5.596752018167751]
  This paper considers estimating the mean outcome of an action from samples obtained in adaptive experiments. In adaptive experiments, the probability of choosing an action is allowed to be sequentially updated based on past observations.
  arXiv  Detail & Related papers  (2021-02-17T19:05:53Z)
• Tracking disease outbreaks from sparse data with Bayesian inference [55.82986443159948]
  The COVID-19 pandemic provides new motivation for estimating the empirical rate of transmission during an outbreak. Standard methods struggle to accommodate the partial observability and sparse data common at finer scales. We propose a Bayesian framework which accommodates partial observability in a principled manner.
  arXiv  Detail & Related papers  (2020-09-12T20:37:33Z)
• Efficient Adaptive Experimental Design for Average Treatment Effect Estimation [20.361745967568154]
  We study how to efficiently estimate average treatment effects (ATEs) using adaptive experiments.<n>In adaptive experiments, experimenters sequentially assign treatments to experimental units while updating treatment probabilities based on past data.<n>We develop nonparametric and nonasymptotic confidence intervals that are valid at any round of the proposed design.
  arXiv  Detail & Related papers  (2020-02-13T02:04:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.