Learning Decision Policies with Instrumental Variables through Double Machine Learning

• URL: http://arxiv.org/abs/2405.08498v3
• Date: Fri, 28 Jun 2024 13:31:48 GMT
• Title: Learning Decision Policies with Instrumental Variables through Double Machine Learning
• Authors: Daqian Shao, Ashkan Soleymani, Francesco Quinzan, Marta Kwiatkowska,
• Abstract summary: A common issue in learning decision-making policies in data-rich settings is spurious correlations in the offline dataset. We propose DML-IV, a non-linear IV regression method that reduces the bias in two-stage IV regressions. It outperforms state-of-the-art IV regression methods on IV regression benchmarks and learns high-performing policies in the presence of instruments.
• Score: 16.842233444365764
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A common issue in learning decision-making policies in data-rich settings is spurious correlations in the offline dataset, which can be caused by hidden confounders. Instrumental variable (IV) regression, which utilises a key unconfounded variable known as the instrument, is a standard technique for learning causal relationships between confounded action, outcome, and context variables. Most recent IV regression algorithms use a two-stage approach, where a deep neural network (DNN) estimator learnt in the first stage is directly plugged into the second stage, in which another DNN is used to estimate the causal effect. Naively plugging the estimator can cause heavy bias in the second stage, especially when regularisation bias is present in the first stage estimator. We propose DML-IV, a non-linear IV regression method that reduces the bias in two-stage IV regressions and effectively learns high-performing policies. We derive a novel learning objective to reduce bias and design the DML-IV algorithm following the double/debiased machine learning (DML) framework. The learnt DML-IV estimator has strong convergence rate and $O(N^{-1/2})$ suboptimality guarantees that match those when the dataset is unconfounded. DML-IV outperforms state-of-the-art IV regression methods on IV regression benchmarks and learns high-performing policies in the presence of instruments.

Related papers

• Automatic debiasing of neural networks via moment-constrained learning [0.0]
  Naively learning the regression function and taking a sample mean of the target functional results in biased estimators. We propose moment-constrained learning as a new RR learning approach that addresses some shortcomings in automatic debiasing.
  arXiv  Detail & Related papers  (2024-09-29T20:56:54Z)
• Geometry-Aware Instrumental Variable Regression [56.16884466478886]
  We propose a transport-based IV estimator that takes into account the geometry of the data manifold through data-derivative information. We provide a simple plug-and-play implementation of our method that performs on par with related estimators in standard settings.
  arXiv  Detail & Related papers  (2024-05-19T17:49:33Z)
• Regularized DeepIV with Model Selection [72.17508967124081]
  Regularized DeepIV (RDIV) regression can converge to the least-norm IV solution. Our method matches the current state-of-the-art convergence rate.
  arXiv  Detail & Related papers  (2024-03-07T05:38:56Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)
• Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization [90.9780151608281]
  In-sample learning (IQL) improves the policy by quantile regression using only data samples. We make a key finding that the in-sample learning paradigm arises under the textitImplicit Value Regularization (IVR) framework. We propose two practical algorithms, Sparse $Q$-learning (EQL) and Exponential $Q$-learning (EQL), which adopt the same value regularization used in existing works.
  arXiv  Detail & Related papers  (2023-03-28T08:30:01Z)
• Confounder Balancing for Instrumental Variable Regression with Latent Variable [29.288045682505615]
  This paper studies the confounding effects from the unmeasured confounders and the imbalance of observed confounders in IV regression. We propose a Confounder Balanced IV Regression (CB-IV) algorithm to remove the bias from the unmeasured confounders and the imbalance of observed confounders.
  arXiv  Detail & Related papers  (2022-11-18T03:13:53Z)
• On Instrumental Variable Regression for Deep Offline Policy Evaluation [37.05492059049681]
  We show that the popular reinforcement learning strategy of estimating the state-action value (Q-function) by minimizing the mean squared Bellman error leads to a regression problem with confounding. We explain why fixing the target Q-network in Deep Q-Networks and Fitted Q Evaluation provides a way of overcoming this confounding. This paper analyzes and compares a wide range of recent IV methods in the context of offline policy evaluation.
  arXiv  Detail & Related papers  (2021-05-21T06:22:34Z)
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning [107.70165026669308]
  In offline reinforcement learning (RL) an optimal policy is learned solely from a priori collected observational data. We study a confounded Markov decision process where the transition dynamics admit an additive nonlinear functional form. We propose a provably efficient IV-aided Value Iteration (IVVI) algorithm based on a primal-dual reformulation of the conditional moment restriction.
  arXiv  Detail & Related papers  (2021-02-19T13:01:40Z)
• Learning Deep Features in Instrumental Variable Regression [42.085253974990046]
  In IV regression, learning proceeds in two stages: stage 1 performs linear regression from the instrument to the treatment; and stage 2 performs linear regression from the treatment to the outcome, conditioned on the instrument. We propose a novel method, deep feature instrumental variable regression (DFIV), to address the case where relations between instruments, treatments, and outcomes may be nonlinear.
  arXiv  Detail & Related papers  (2020-10-14T15:14:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.