Evaluating the Effectiveness of Index-Based Treatment Allocation
- URL: http://arxiv.org/abs/2402.11771v1
- Date: Mon, 19 Feb 2024 01:55:55 GMT
- Title: Evaluating the Effectiveness of Index-Based Treatment Allocation
- Authors: Niclas Boehmer, Yash Nair, Sanket Shah, Lucas Janson, Aparna Taneja,
Milind Tambe
- Abstract summary: When resources are scarce, an allocation policy is needed to decide who receives a resource.
This paper introduces methods to evaluate index-based allocation policies using data from a randomized control trial.
- Score: 42.040099398176665
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: When resources are scarce, an allocation policy is needed to decide who
receives a resource. This problem occurs, for instance, when allocating scarce
medical resources and is often solved using modern ML methods. This paper
introduces methods to evaluate index-based allocation policies -- that allocate
a fixed number of resources to those who need them the most -- by using data
from a randomized control trial. Such policies create dependencies between
agents, which render the assumptions behind standard statistical tests invalid
and limit the effectiveness of estimators. Addressing these challenges, we
translate and extend recent ideas from the statistics literature to present an
efficient estimator and methods for computing asymptotically correct confidence
intervals. This enables us to effectively draw valid statistical conclusions, a
critical gap in previous work. Our extensive experiments validate our
methodology in practical settings, while also showcasing its statistical power.
We conclude by proposing and empirically verifying extensions of our
methodology that enable us to reevaluate a past randomized control trial to
evaluate different ML allocation policies in the context of a mHealth program,
drawing previously invisible conclusions.
Related papers
- Source-Free Domain-Invariant Performance Prediction [68.39031800809553]
We propose a source-free approach centred on uncertainty-based estimation, using a generative model for calibration in the absence of source data.
Our experiments on benchmark object recognition datasets reveal that existing source-based methods fall short with limited source sample availability.
Our approach significantly outperforms the current state-of-the-art source-free and source-based methods, affirming its effectiveness in domain-invariant performance estimation.
arXiv Detail & Related papers (2024-08-05T03:18:58Z) - Targeted Machine Learning for Average Causal Effect Estimation Using the
Front-Door Functional [3.0232957374216953]
evaluating the average causal effect (ACE) of a treatment on an outcome often involves overcoming the challenges posed by confounding factors in observational studies.
Here, we introduce novel estimation strategies for the front-door criterion based on the targeted minimum loss-based estimation theory.
We demonstrate the applicability of these estimators to analyze the effect of early stage academic performance on future yearly income.
arXiv Detail & Related papers (2023-12-15T22:04:53Z) - Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning.
Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
arXiv Detail & Related papers (2023-03-11T11:42:26Z) - Improved Policy Evaluation for Randomized Trials of Algorithmic Resource
Allocation [54.72195809248172]
We present a new estimator leveraging our proposed novel concept, that involves retrospective reshuffling of participants across experimental arms at the end of an RCT.
We prove theoretically that such an estimator is more accurate than common estimators based on sample means.
arXiv Detail & Related papers (2023-02-06T05:17:22Z) - Variance-Optimal Augmentation Logging for Counterfactual Evaluation in
Contextual Bandits [25.153656462604268]
Methods for offline A/B testing and counterfactual learning are seeing rapid adoption in search and recommender systems.
The counterfactual estimators that are commonly used in these methods can have large bias and large variance when the logging policy is very different from the target policy being evaluated.
This paper introduces Minimum Variance Augmentation Logging (MVAL), a method for constructing logging policies that minimize the variance of the downstream evaluation or learning problem.
arXiv Detail & Related papers (2022-02-03T17:37:11Z) - Statistical Bootstrapping for Uncertainty Estimation in Off-Policy
Evaluation [38.31971190670345]
We investigate the potential for statistical bootstrapping to be used as a way to produce calibrated confidence intervals for the true value of the policy.
We show that it can yield accurate confidence intervals in a variety of conditions, including challenging continuous control environments and small data regimes.
arXiv Detail & Related papers (2020-07-27T14:49:22Z) - Distributionally Robust Batch Contextual Bandits [20.667213458836734]
Policy learning using historical observational data is an important problem that has found widespread applications.
Existing literature rests on the crucial assumption that the future environment where the learned policy will be deployed is the same as the past environment.
In this paper, we lift this assumption and aim to learn a distributionally robust policy with incomplete observational data.
arXiv Detail & Related papers (2020-06-10T03:11:40Z) - Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation [49.502277468627035]
This paper studies the statistical theory of batch data reinforcement learning with function approximation.
Consider the off-policy evaluation problem, which is to estimate the cumulative value of a new target policy from logged history.
arXiv Detail & Related papers (2020-02-21T19:20:57Z) - Efficient Policy Learning from Surrogate-Loss Classification Reductions [65.91730154730905]
We consider the estimation problem given by a weighted surrogate-loss classification reduction of policy learning.
We show that, under a correct specification assumption, the weighted classification formulation need not be efficient for policy parameters.
We propose an estimation approach based on generalized method of moments, which is efficient for the policy parameters.
arXiv Detail & Related papers (2020-02-12T18:54:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.