Benchmarking Estimators for Natural Experiments: A Novel Dataset and a Doubly Robust Algorithm

• URL: http://arxiv.org/abs/2409.04500v1
• Date: Fri, 6 Sep 2024 15:44:45 GMT
• Title: Benchmarking Estimators for Natural Experiments: A Novel Dataset and a Doubly Robust Algorithm
• Authors: R. Teal Witter, Christopher Musco,
• Abstract summary: We introduce a novel natural experiment dataset obtained from an early childhood literacy nonprofit. Applying over 20 established estimators to the dataset produces inconsistent results in evaluating the nonprofit's efficacy. We create a benchmark to evaluate estimator accuracy using synthetic outcomes.
• Score: 12.201705893125775
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating the effect of treatments from natural experiments, where treatments are pre-assigned, is an important and well-studied problem. We introduce a novel natural experiment dataset obtained from an early childhood literacy nonprofit. Surprisingly, applying over 20 established estimators to the dataset produces inconsistent results in evaluating the nonprofit's efficacy. To address this, we create a benchmark to evaluate estimator accuracy using synthetic outcomes, whose design was guided by domain experts. The benchmark extensively explores performance as real world conditions like sample size, treatment correlation, and propensity score accuracy vary. Based on our benchmark, we observe that the class of doubly robust treatment effect estimators, which are based on simple and intuitive regression adjustment, generally outperform other more complicated estimators by orders of magnitude. To better support our theoretical understanding of doubly robust estimators, we derive a closed form expression for the variance of any such estimator that uses dataset splitting to obtain an unbiased estimate. This expression motivates the design of a new doubly robust estimator that uses a novel loss function when fitting functions for regression adjustment. We release the dataset and benchmark in a Python package; the package is built in a modular way to facilitate new datasets and estimators.

Related papers

• Heavy-tailed Contamination is Easier than Adversarial Contamination [8.607294463464523]
  A body of work in the statistics and computer science communities dating back to Huber (Huber, 1960) has led to statistically and computationally efficient outlier-robust estimators. Two particular outlier models have received significant attention: the adversarial and heavy-tailed models.
  arXiv  Detail & Related papers  (2024-11-22T19:00:33Z)
• Efficient adjustment for complex covariates: Gaining efficiency with DOPE [56.537164957672715]
  We propose a framework that accommodates adjustment for any subset of information expressed by the covariates. Based on our theoretical results, we propose the Debiased Outcome-adapted Propensity Estorimator (DOPE) for efficient estimation of the average treatment effect (ATE) Our results show that the DOPE provides an efficient and robust methodology for ATE estimation in various observational settings.
  arXiv  Detail & Related papers  (2024-02-20T13:02:51Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Semi-Supervised Quantile Estimation: Robust and Efficient Inference in High Dimensional Settings [0.5735035463793009]
  We consider quantile estimation in a semi-supervised setting, characterized by two available data sets. We propose a family of semi-supervised estimators for the response quantile(s) based on the two data sets.
  arXiv  Detail & Related papers  (2022-01-25T10:02:23Z)
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model [78.55482897452417]
  We aim at improving data efficiency for both classification and regression setups in deep learning. To take the power of both worlds, we propose a novel X-model. X-model plays a minimax game between the feature extractor and task-specific heads.
  arXiv  Detail & Related papers  (2021-10-09T13:56:48Z)
• Near-optimal inference in adaptive linear regression [60.08422051718195]
  Even simple methods like least squares can exhibit non-normal behavior when data is collected in an adaptive manner. We propose a family of online debiasing estimators to correct these distributional anomalies in at least squares estimation. We demonstrate the usefulness of our theory via applications to multi-armed bandit, autoregressive time series estimation, and active learning with exploration.
  arXiv  Detail & Related papers  (2021-07-05T21:05:11Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Enhanced Doubly Robust Learning for Debiasing Post-click Conversion Rate Estimation [29.27760413892272]
  Post-click conversion, as a strong signal indicating the user preference, is salutary for building recommender systems. Currently, most existing methods utilize counterfactual learning to debias recommender systems. We propose a novel double learning approach for the MRDR estimator, which can convert the error imputation into the general CVR estimation.
  arXiv  Detail & Related papers  (2021-05-28T06:59:49Z)
• Higher-Order Orthogonal Causal Learning for Treatment Effect [15.652550362252205]
  We present an algorithm that enables us to obtain the debiased estimator recovered from the score function. We also undergo comprehensive experiments to test the power of the estimator we construct from the score function using both the simulated datasets and the real datasets.
  arXiv  Detail & Related papers  (2021-03-22T14:04:13Z)
• Semi-supervised learning and the question of true versus estimated propensity scores [0.456877715768796]
  We propose a simple procedure that reconciles the strong intuition that a known propensity functions should be useful for estimating treatment effects. Further, simulation studies suggest that direct regression may be preferable to inverse-propensity weight estimators in many circumstances.
  arXiv  Detail & Related papers  (2020-09-14T04:13:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.