Efficient Multi-Cohort Inference for Long-Term Effects and Lifetime Value in A/B Testing with User Learning

• URL: http://arxiv.org/abs/2604.20777v1
• Date: Wed, 22 Apr 2026 17:05:11 GMT
• Title: Efficient Multi-Cohort Inference for Long-Term Effects and Lifetime Value in A/B Testing with User Learning
• Abstract summary: In streaming platforms churn is extremely costly, yet A/B tests are typically evaluated using outcomes observed within a limited experimental horizon.<n>We introduce a method that estimates long-term treatment effects (LTE) and residual lifetime value change ($ERLV$) in short multi-cohort A/B tests under user learning.<n>Our framework enables simultaneous evaluation of steady-state impact and residual user value within a single experiment.
• Score: 1.9775461158626977
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In streaming platforms churn is extremely costly, yet A/B tests are typically evaluated using outcomes observed within a limited experimental horizon. Even when both short- and predicted long-term engagement metrics are considered, they may fail to capture how a treatment affects users' retention. Consequently, an intervention may appear beneficial in the short term and neutral in the long term while still generating lower total value than the control due to users churn. To address this limitation, we introduce a method that estimates long-term treatment effects (LTE) and residual lifetime value change ($ΔERLV$) in short multi-cohort A/B tests under user learning. To estimate time-varying treatment effects efficiently, we introduce an inverse-variance weighted estimator that combines multiple cohorts estimates, reducing variance relative to standard approaches in the literature. The estimated treatment trajectory is then modeled as a parametric decay to recover both the asymptotic treatment effect and the cumulative value generated over time. Our framework enables simultaneous evaluation of steady-state impact and residual user value within a single experiment. Empirical results show improved precision in estimating LTE and $ΔERLV$ and identify scenarios in which relying on either short-term or long-term metrics alone would lead to incorrect product decisions.

Related papers

• Estimating heterogeneous treatment effects with survival outcomes via a deep survival learner [5.540352966241158]
  Estimating heterogeneous treatment effects in survival settings is complicated by right censoring and the time-varying nature of the estimand.<n>We propose a deep survival learner for estimating heterogeneous treatment effects with right-censored outcomes.
  arXiv  Detail & Related papers  (2026-04-12T01:11:44Z)
• Orthogonal Learner for Estimating Heterogeneous Long-Term Treatment Effects [56.15663253912398]
  We introduce the LT-O-learners (Long-Term Orthogonal learners) for HLTE estimation.<n>The learners are designed for the canonical HLTE setting that combines a short-term randomized dataset with a long-term historical dataset.<n>We show that the learners are robust to errors in the nuisance estimation.
  arXiv  Detail & Related papers  (2026-04-01T13:56:19Z)
• Targeted learning of heterogeneous treatment effect curves for right censored or left truncated time-to-event data [1.360712433091342]
  We introduce surv-iTMLE, a learning procedure for estimating the difference in the conditional survival probabilities under two treatments.<n>Unlike existing estimators, surv-iTMLE accommodates both left truncation and right censoring while enforcing smoothness and boundedness of the estimated treatment effect curve over time.<n>We show that surv-iTMLE outperforms existing methods in terms of bias and smoothness of time-varying effect estimates in finite samples.
  arXiv  Detail & Related papers  (2026-03-27T15:08:07Z)
• Improving the Estimation of Lifetime Effects in A/B Testing via Treatment Locality [16.36651676133996]
  We develop optimal inference techniques for general A/B testing in Markov Decision Processes.<n>We propose methods to harness the localized structure by sharing information on the non-targeted states.<n>We show that all such estimators can benefit from variance reduction through information sharing without increasing their bias.
  arXiv  Detail & Related papers  (2024-07-29T00:41:11Z)
• Estimating Long-term Heterogeneous Dose-response Curve: Generalization Bound Leveraging Optimal Transport Weights [54.65531750162626]
  Long-term treatment effect estimation is a significant but challenging problem in many applications.<n>Existing methods rely on ideal assumptions, such as no unobserved confounders or binary treatment, to estimate long-term average treatment effects.<n>We introduce an optimal transport weighting framework to align the long-term observational data to an auxiliary short-term experimental data.
  arXiv  Detail & Related papers  (2024-06-27T14:13:46Z)
• Long-term Off-Policy Evaluation and Learning [21.047613223586794]
  Short- and long-term outcomes of an algorithm often differ, with damaging downstream effects. It takes months or even longer to observe the long-term outcomes of interest, making the algorithm selection process unacceptably slow. We propose a new framework called Long-term Off-Policy Evaluation (LOPE), which is based on reward function decomposition.
  arXiv  Detail & Related papers  (2024-04-24T06:59:59Z)
• Choosing a Proxy Metric from Past Experiments [54.338884612982405]
  In many randomized experiments, the treatment effect of the long-term metric is often difficult or infeasible to measure. A common alternative is to measure several short-term proxy metrics in the hope they closely track the long-term metric. We introduce a new statistical framework to both define and construct an optimal proxy metric for use in a homogeneous population of randomized experiments.
  arXiv  Detail & Related papers  (2023-09-14T17:43:02Z)
• Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation [54.72195809248172]
  We present a new estimator leveraging our proposed novel concept, that involves retrospective reshuffling of participants across experimental arms at the end of an RCT. We prove theoretically that such an estimator is more accurate than common estimators based on sample means.
  arXiv  Detail & Related papers  (2023-02-06T05:17:22Z)
• TCFimt: Temporal Counterfactual Forecasting from Individual Multiple Treatment Perspective [50.675845725806724]
  We propose a comprehensive framework of temporal counterfactual forecasting from an individual multiple treatment perspective (TCFimt) TCFimt constructs adversarial tasks in a seq2seq framework to alleviate selection and time-varying bias and designs a contrastive learning-based block to decouple a mixed treatment effect into separated main treatment effects and causal interactions. The proposed method shows satisfactory performance in predicting future outcomes with specific treatments and in choosing optimal treatment type and timing than state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-12-17T15:01:05Z)
• A Reinforcement Learning Approach to Estimating Long-term Treatment Effects [13.371851720834918]
  A limitation with randomized experiments is that they do not easily extend to measure long-term effects. We take a reinforcement learning (RL) approach that estimates the average reward in a Markov process. Motivated by real-world scenarios where the observed state transition is nonstationary, we develop a new algorithm for a class of nonstationary problems.
  arXiv  Detail & Related papers  (2022-10-14T05:33:19Z)
• Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score [59.75318183140857]
  Estimating causal effects from randomized experiments is central to clinical research. Most methods for historical borrowing achieve reductions in variance by sacrificing strict type-I error rate control.
  arXiv  Detail & Related papers  (2020-12-17T21:10:10Z)
• On the role of surrogates in the efficient estimation of treatment effects with limited outcome data [43.17788100119767]
  We study how incorporating data on units for which only surrogate outcomes not of primary interest are observed can increase the precision of ATE estimation. We develop robust ATE estimation and inference methods that realize these efficiency gains.
  arXiv  Detail & Related papers  (2020-03-27T13:31:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.