Conformal Off-Policy Prediction

• URL: http://arxiv.org/abs/2206.06711v1
• Date: Tue, 14 Jun 2022 09:31:18 GMT
• Title: Conformal Off-Policy Prediction
• Authors: Yingying Zhang, Chengchun Shi, Shikai Luo
• Abstract summary: We develop a novel procedure to produce reliable interval estimators for a target policy's return starting from any initial state. Our main idea lies in designing a pseudo policy that generates subsamples as if they were sampled from the target policy.
• Score: 14.83348592874271
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Off-policy evaluation is critical in a number of applications where new policies need to be evaluated offline before online deployment. Most existing methods focus on the expected return, define the target parameter through averaging and provide a point estimator only. In this paper, we develop a novel procedure to produce reliable interval estimators for a target policy's return starting from any initial state. Our proposal accounts for the variability of the return around its expectation, focuses on the individual effect and offers valid uncertainty quantification. Our main idea lies in designing a pseudo policy that generates subsamples as if they were sampled from the target policy so that existing conformal prediction algorithms are applicable to prediction interval construction. Our methods are justified by theories, synthetic data and real data from short-video platforms.

Related papers

• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Quantile Off-Policy Evaluation via Deep Conditional Generative Learning [21.448553360543478]
  Off-Policy evaluation (OPE) is concerned with evaluating a new target policy using offline data generated by a potentially different behavior policy. We propose a doubly-robust inference procedure for quantile OPE in sequential decision making. We demonstrate the advantages of this proposed estimator through both simulations and a real-world dataset from a short-video platform.
  arXiv  Detail & Related papers  (2022-12-29T22:01:43Z)
• Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation [60.71312668265873]
  We develop a method to balance the need for personalization with confident predictions. We show that our method can be used to form accurate predictions of heterogeneous treatment effects.
  arXiv  Detail & Related papers  (2021-11-28T23:19:12Z)
• Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
  In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy. We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
  arXiv  Detail & Related papers  (2020-11-08T23:16:19Z)
• Distributionally Robust Batch Contextual Bandits [20.667213458836734]
  Policy learning using historical observational data is an important problem that has found widespread applications. Existing literature rests on the crucial assumption that the future environment where the learned policy will be deployed is the same as the past environment. In this paper, we lift this assumption and aim to learn a distributionally robust policy with incomplete observational data.
  arXiv  Detail & Related papers  (2020-06-10T03:11:40Z)
• Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies [80.42316902296832]
  We study the estimation of policy value and gradient of a deterministic policy from off-policy data when actions are continuous. In this setting, standard importance sampling and doubly robust estimators for policy value and gradient fail because the density ratio does not exist. We propose several new doubly robust estimators based on different kernelization approaches.
  arXiv  Detail & Related papers  (2020-06-06T15:52:05Z)
• Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation [49.502277468627035]
  This paper studies the statistical theory of batch data reinforcement learning with function approximation. Consider the off-policy evaluation problem, which is to estimate the cumulative value of a new target policy from logged history.
  arXiv  Detail & Related papers  (2020-02-21T19:20:57Z)
• Statistically Efficient Off-Policy Policy Gradients [80.42316902296832]
  We consider the statistically efficient estimation of policy gradients from off-policy data. We propose a meta-algorithm that achieves the lower bound without any parametric assumptions. We establish guarantees on the rate at which we approach a stationary point when we take steps in the direction of our new estimated policy gradient.
  arXiv  Detail & Related papers  (2020-02-10T18:41:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.