Regularized DeepIV with Model Selection
- URL: http://arxiv.org/abs/2403.04236v1
- Date: Thu, 7 Mar 2024 05:38:56 GMT
- Title: Regularized DeepIV with Model Selection
- Authors: Zihao Li, Hui Lan, Vasilis Syrgkanis, Mengdi Wang, Masatoshi Uehara
- Abstract summary: Regularized DeepIV (RDIV) regression can converge to the least-norm IV solution.
Our method matches the current state-of-the-art convergence rate.
- Score: 72.17508967124081
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: In this paper, we study nonparametric estimation of instrumental variable
(IV) regressions. While recent advancements in machine learning have introduced
flexible methods for IV estimation, they often encounter one or more of the
following limitations: (1) restricting the IV regression to be uniquely
identified; (2) requiring minimax computation oracle, which is highly unstable
in practice; (3) absence of model selection procedure. In this paper, we
present the first method and analysis that can avoid all three limitations,
while still enabling general function approximation. Specifically, we propose a
minimax-oracle-free method called Regularized DeepIV (RDIV) regression that can
converge to the least-norm IV solution. Our method consists of two stages:
first, we learn the conditional distribution of covariates, and by utilizing
the learned distribution, we learn the estimator by minimizing a
Tikhonov-regularized loss function. We further show that our method allows
model selection procedures that can achieve the oracle rates in the
misspecified regime. When extended to an iterative estimator, our method
matches the current state-of-the-art convergence rate. Our method is a Tikhonov
regularized variant of the popular DeepIV method with a non-parametric MLE
first-stage estimator, and our results provide the first rigorous guarantees
for this empirically used method, showcasing the importance of regularization
which was absent from the original work.
Related papers
- Learning Decision Policies with Instrumental Variables through Double Machine Learning [16.842233444365764]
A common issue in learning decision-making policies in data-rich settings is spurious correlations in the offline dataset.
We propose DML-IV, a non-linear IV regression method that reduces the bias in two-stage IV regressions.
It outperforms state-of-the-art IV regression methods on IV regression benchmarks and learns high-performing policies in the presence of instruments.
arXiv Detail & Related papers (2024-05-14T10:55:04Z) - Nonparametric Instrumental Variable Regression through Stochastic Approximate Gradients [0.3277163122167434]
We show how to formulate a functional gradient descent algorithm to tackle NPIV regression by directly minimizing the populational risk.
We provide theoretical support in the form of bounds on the excess risk, and conduct numerical experiments showcasing our method's superior stability and competitive performance.
This algorithm enables flexible estimator choices, such as neural networks or kernel based methods, as well as non-quadratic loss functions.
arXiv Detail & Related papers (2024-02-08T12:50:38Z) - Minimax Instrumental Variable Regression and $L_2$ Convergence
Guarantees without Identification or Closedness [71.42652863687117]
We study nonparametric estimation of instrumental variable (IV) regressions.
We propose a new penalized minimax estimator that can converge to a fixed IV solution.
We derive a strong $L$ error rate for our estimator under lax conditions.
arXiv Detail & Related papers (2023-02-10T18:08:49Z) - Confounder Balancing for Instrumental Variable Regression with Latent
Variable [29.288045682505615]
This paper studies the confounding effects from the unmeasured confounders and the imbalance of observed confounders in IV regression.
We propose a Confounder Balanced IV Regression (CB-IV) algorithm to remove the bias from the unmeasured confounders and the imbalance of observed confounders.
arXiv Detail & Related papers (2022-11-18T03:13:53Z) - Vector-Valued Least-Squares Regression under Output Regularity
Assumptions [73.99064151691597]
We propose and analyse a reduced-rank method for solving least-squares regression problems with infinite dimensional output.
We derive learning bounds for our method, and study under which setting statistical performance is improved in comparison to full-rank method.
arXiv Detail & Related papers (2022-11-16T15:07:00Z) - Variance Reduction for Policy-Gradient Methods via Empirical Variance
Minimization [69.32510868632988]
Policy-gradient methods in Reinforcement Learning suffer from the high variance of the gradient estimate.
In this paper we for the first time investigate the performance of the one called Empirical Variance(EV)
Our experiments indicate that in terms of variance reduction EV-based methods are much better than A2C and allow stronger variance reduction.
arXiv Detail & Related papers (2022-06-14T13:18:49Z) - Near-optimal inference in adaptive linear regression [60.08422051718195]
Even simple methods like least squares can exhibit non-normal behavior when data is collected in an adaptive manner.
We propose a family of online debiasing estimators to correct these distributional anomalies in at least squares estimation.
We demonstrate the usefulness of our theory via applications to multi-armed bandit, autoregressive time series estimation, and active learning with exploration.
arXiv Detail & Related papers (2021-07-05T21:05:11Z) - Maximum Moment Restriction for Instrumental Variable Regression [23.785259180682004]
We propose a simple framework for nonlinear instrumental variable (IV) regression based on a kernelized conditional moment restriction (CMR)
The MMR is formulated by maximizing the interaction between the residual and the instruments belonging to a unit ball in a kernel reproducing space (RKHS)
We demonstrate the advantages of our framework over existing ones using experiments on both synthetic and real-world data.
arXiv Detail & Related papers (2020-10-15T12:02:25Z) - Regret minimization in stochastic non-convex learning via a
proximal-gradient approach [80.59047515124198]
Motivated by applications in machine learning and operations, we regret with first-order oracle feedback minimization online constrained problems.
We develop a new prox-grad with guarantees proximal complexity reduction techniques.
arXiv Detail & Related papers (2020-10-13T09:22:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.