Assessment of Treatment Effect Estimators for Heavy-Tailed Data
- URL: http://arxiv.org/abs/2112.07602v1
- Date: Tue, 14 Dec 2021 17:53:01 GMT
- Title: Assessment of Treatment Effect Estimators for Heavy-Tailed Data
- Authors: Nilesh Tripuraneni, Dhruv Madeka, Dean Foster, Dominique
Perrault-Joncas, Michael I. Jordan
- Abstract summary: A central obstacle in the objective assessment of treatment effect (TE) estimators in randomized control trials (RCTs) is the lack of ground truth (or validation set) to test their performance.
We provide a novel cross-validation-like methodology to address this challenge.
We evaluate our methodology across 709 RCTs implemented in the Amazon supply chain.
- Score: 70.72363097550483
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: A central obstacle in the objective assessment of treatment effect (TE)
estimators in randomized control trials (RCTs) is the lack of ground truth (or
validation set) to test their performance. In this paper, we provide a novel
cross-validation-like methodology to address this challenge. The key insight of
our procedure is that the noisy (but unbiased) difference-of-means estimate can
be used as a ground truth "label" on a portion of the RCT, to test the
performance of an estimator trained on the other portion. We combine this
insight with an aggregation scheme, which borrows statistical strength across a
large collection of RCTs, to present an end-to-end methodology for judging an
estimator's ability to recover the underlying treatment effect. We evaluate our
methodology across 709 RCTs implemented in the Amazon supply chain. In the
corpus of AB tests at Amazon, we highlight the unique difficulties associated
with recovering the treatment effect due to the heavy-tailed nature of the
response variables. In this heavy-tailed setting, our methodology suggests that
procedures that aggressively downweight or truncate large values, while
introducing bias lower the variance enough to ensure that the treatment effect
is more accurately estimated.
Related papers
- Training and Validating a Treatment Recommender with Partial Verification Evidence [1.0693162404690828]
Current clinical decision support systems (DSS) are trained and validated on observational data from the target clinic.
This is problematic for treatments validated in a randomized clinical trial (RCT) but not yet introduced in any clinic.
Key challenges we address are of missingness -- missing rationale for treatment assignment (the assignment is at random), and missing verification evidence.
arXiv Detail & Related papers (2024-06-10T09:23:00Z) - Adaptive-TMLE for the Average Treatment Effect based on Randomized Controlled Trial Augmented with Real-World Data [0.0]
We consider the problem of estimating the average treatment effect (ATE) when both randomized control trial (RCT) data and real-world data (RWD) are available.
We introduce an adaptive targeted minimum loss-based estimation framework to estimate them.
arXiv Detail & Related papers (2024-05-12T07:10:26Z) - A Weighted Prognostic Covariate Adjustment Method for Efficient and
Powerful Treatment Effect Inferences in Randomized Controlled Trials [0.28087862620958753]
A crucial task for a randomized controlled trial (RCT) is to specify a statistical method that can yield an efficient estimator and powerful test for the treatment effect.
Training a generative AI algorithm on historical control data enables one to construct a digital twin generator (DTG) for RCT participants.
DTG generates a probability distribution for RCT participants' potential control outcome.
arXiv Detail & Related papers (2023-09-25T16:14:13Z) - Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment
Effect Estimation [137.3520153445413]
A notable gap exists in the evaluation of causal discovery methods, where insufficient emphasis is placed on downstream inference.
We evaluate seven established baseline causal discovery methods including a newly proposed method based on GFlowNets.
The results of our study demonstrate that some of the algorithms studied are able to effectively capture a wide range of useful and diverse ATE modes.
arXiv Detail & Related papers (2023-07-11T02:58:10Z) - B-Learner: Quasi-Oracle Bounds on Heterogeneous Causal Effects Under
Hidden Confounding [51.74479522965712]
We propose a meta-learner called the B-Learner, which can efficiently learn sharp bounds on the CATE function under limits on hidden confounding.
We prove its estimates are valid, sharp, efficient, and have a quasi-oracle property with respect to the constituent estimators under more general conditions than existing methods.
arXiv Detail & Related papers (2023-04-20T18:07:19Z) - Improved Policy Evaluation for Randomized Trials of Algorithmic Resource
Allocation [54.72195809248172]
We present a new estimator leveraging our proposed novel concept, that involves retrospective reshuffling of participants across experimental arms at the end of an RCT.
We prove theoretically that such an estimator is more accurate than common estimators based on sample means.
arXiv Detail & Related papers (2023-02-06T05:17:22Z) - Robust and Agnostic Learning of Conditional Distributional Treatment
Effects [62.44901952244514]
The conditional average treatment effect (CATE) is the best point prediction of individual causal effects.
In aggregate analyses, this is usually addressed by measuring distributional treatment effect (DTE)
We provide a new robust and model-agnostic methodology for learning the conditional DTE (CDTE) for a wide class of problems.
arXiv Detail & Related papers (2022-05-23T17:40:31Z) - Treatment Effect Risk: Bounds and Inference [58.442274475425144]
Since the average treatment effect measures the change in social welfare, even if positive, there is a risk of negative effect on, say, some 10% of the population.
In this paper we consider how to nonetheless assess this important risk measure, formalized as the conditional value at risk (CVaR) of the ITE distribution.
Some bounds can also be interpreted as summarizing a complex CATE function into a single metric and are of interest independently of being a bound.
arXiv Detail & Related papers (2022-01-15T17:21:26Z) - Weighting-Based Treatment Effect Estimation via Distribution Learning [14.438302755258547]
We develop a distribution learning-based weighting method for treatment effect estimation.
Our method outperforms several cutting-edge weighting-only benchmarking methods.
It maintains its advantage under a doubly-robust estimation framework.
arXiv Detail & Related papers (2020-12-26T20:15:44Z) - Confounding Feature Acquisition for Causal Effect Estimation [6.174721516017138]
We frame this challenge as a problem of feature acquisition of confounding features for causal inference.
Our goal is to prioritize acquiring values for a fixed and known subset of missing confounders in samples that lead to efficient average treatment effect estimation.
arXiv Detail & Related papers (2020-11-17T16:28:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.