Hyperparameter Tuning for Causal Inference with Double Machine Learning:
A Simulation Study
- URL: http://arxiv.org/abs/2402.04674v1
- Date: Wed, 7 Feb 2024 09:01:51 GMT
- Title: Hyperparameter Tuning for Causal Inference with Double Machine Learning:
A Simulation Study
- Authors: Philipp Bach and Oliver Schacht and Victor Chernozhukov and Sven
Klaassen and Martin Spindler
- Abstract summary: We empirically assess the relationship between the predictive performance of machine learning methods and the resulting causal estimation.
We conduct an extensive simulation study using data from the 2019 Atlantic Causal Inference Conference Data Challenge.
- Score: 4.526082390949313
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Proper hyperparameter tuning is essential for achieving optimal performance
of modern machine learning (ML) methods in predictive tasks. While there is an
extensive literature on tuning ML learners for prediction, there is only little
guidance available on tuning ML learners for causal machine learning and how to
select among different ML learners. In this paper, we empirically assess the
relationship between the predictive performance of ML methods and the resulting
causal estimation based on the Double Machine Learning (DML) approach by
Chernozhukov et al. (2018). DML relies on estimating so-called nuisance
parameters by treating them as supervised learning problems and using them as
plug-in estimates to solve for the (causal) parameter. We conduct an extensive
simulation study using data from the 2019 Atlantic Causal Inference Conference
Data Challenge. We provide empirical insights on the role of hyperparameter
tuning and other practical decisions for causal estimation with DML. First, we
assess the importance of data splitting schemes for tuning ML learners within
Double Machine Learning. Second, we investigate how the choice of ML methods
and hyperparameters, including recent AutoML frameworks, impacts the estimation
performance for a causal parameter of interest. Third, we assess to what extent
the choice of a particular causal model, as characterized by incorporated
parametric assumptions, can be based on predictive performance metrics.
Related papers
- Verbalized Machine Learning: Revisiting Machine Learning with Language Models [63.10391314749408]
We introduce the framework of verbalized machine learning (VML)
VML constrains the parameter space to be human-interpretable natural language.
We conduct several studies to empirically evaluate the effectiveness of VML.
arXiv Detail & Related papers (2024-06-06T17:59:56Z) - C-Learner: Constrained Learning for Causal Inference and Semiparametric Statistics [5.395560682099634]
We present a novel correction method that solves for the best plug-in estimator under the constraint that the first-order error of the estimator with respect to the nuisance parameter estimate is zero.
Our semi inference approach, which we call the "C-Learner", can be implemented with modern machine learning methods such as neural networks and tree ensembles.
arXiv Detail & Related papers (2024-05-15T16:38:28Z) - Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective [106.92016199403042]
We empirically investigate knowledge transfer from larger to smaller models through a parametric perspective.
We employ sensitivity-based techniques to extract and align knowledge-specific parameters between different large language models.
Our findings highlight the critical factors contributing to the process of parametric knowledge transfer.
arXiv Detail & Related papers (2023-10-17T17:58:34Z) - Adaptive debiased machine learning using data-driven model selection
techniques [0.5735035463793007]
Adaptive Debiased Machine Learning (ADML) is a nonbiased framework that combines data-driven model selection and debiased machine learning techniques.
ADML avoids the bias introduced by model misspecification and remains free from the restrictions of parametric and semi models.
We provide a broad class of ADML estimators for estimating the average treatment effect in adaptive partially linear regression models.
arXiv Detail & Related papers (2023-07-24T06:16:17Z) - To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs.
We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting.
Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
arXiv Detail & Related papers (2023-05-22T17:02:15Z) - Hyperparameter Tuning and Model Evaluation in Causal Effect Estimation [2.7823528791601686]
This paper investigates the interplay between the four different aspects of model evaluation for causal effect estimation.
We find that most causal estimators are roughly equivalent in performance if tuned thoroughly enough.
We call for more research into causal model evaluation to unlock the optimum performance not currently being delivered even by state-of-the-art procedures.
arXiv Detail & Related papers (2023-03-02T17:03:02Z) - Transfer Learning without Knowing: Reprogramming Black-box Machine
Learning Models with Scarce Data and Limited Resources [78.72922528736011]
We propose a novel approach, black-box adversarial reprogramming (BAR), that repurposes a well-trained black-box machine learning model.
Using zeroth order optimization and multi-label mapping techniques, BAR can reprogram a black-box ML model solely based on its input-output responses.
BAR outperforms state-of-the-art methods and yields comparable performance to the vanilla adversarial reprogramming method.
arXiv Detail & Related papers (2020-07-17T01:52:34Z) - A Semiparametric Approach to Interpretable Machine Learning [9.87381939016363]
Black box models in machine learning have demonstrated excellent predictive performance in complex problems and high-dimensional settings.
Their lack of transparency and interpretability restrict the applicability of such models in critical decision-making processes.
We propose a novel approach to trading off interpretability and performance in prediction models using ideas from semiparametric statistics.
arXiv Detail & Related papers (2020-06-08T16:38:15Z) - Machine learning for causal inference: on the use of cross-fit
estimators [77.34726150561087]
Doubly-robust cross-fit estimators have been proposed to yield better statistical properties.
We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE)
When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
arXiv Detail & Related papers (2020-04-21T23:09:55Z) - Localized Debiased Machine Learning: Efficient Inference on Quantile
Treatment Effects and Beyond [69.83813153444115]
We consider an efficient estimating equation for the (local) quantile treatment effect ((L)QTE) in causal inference.
Debiased machine learning (DML) is a data-splitting approach to estimating high-dimensional nuisances.
We propose localized debiased machine learning (LDML), which avoids this burdensome step.
arXiv Detail & Related papers (2019-12-30T14:42:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.