Logging Policy Design for Off-Policy Evaluation

• URL: http://arxiv.org/abs/2605.15108v2
• Date: Fri, 15 May 2026 16:30:03 GMT
• Title: Logging Policy Design for Off-Policy Evaluation
• Abstract summary: Off-policy evaluation (OPE) estimates the value of a target treatment policy using data collected by a different logging policy.<n>We study how to design logging policies that minimize OPE error for given target policies.
• Score: 0.509780930114934
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Off-policy evaluation (OPE) estimates the value of a target treatment policy (e.g., a recommender system) using data collected by a different logging policy. It enables high-stakes experimentation without live deployment, yet in practice accuracy depends heavily on the logging policy used to collect data for computing the estimate. We study how to design logging policies that minimize OPE error for given target policies. We characterize a fundamental reward-coverage tradeoff: concentrating probability mass on high-reward actions reduces variance but risks missing signal on actions the target policy may take. We propose a unifying framework for logging policy design and derive optimal policies in canonical informational regimes where the target policy and reward distribution are (i) known, (ii) unknown, and (iii) partially known through priors or noisy estimates at logging time. Our results provide actionable guidance for firms choosing among multiple candidate recommendation systems. We demonstrate the importance of treatment selection when gathering data for OPE, and describe theoretically optimal approaches when this is a firm's primary objective. We also distill practical design principles for selecting logging policies when operational constraints prevent implementing the theoretical optimum.

Related papers

• Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies [17.672063433232196]
  Off-Policy Evaluation (OPE) is an important practical problem in ranking systems.<n>The goal is to estimate the expected performance of a new ranking policy using only logged offline data collected under a different, logging policy.<n>We propose novel estimators, Click-based Inverse Propensity Score (CIPS), exploiting the intrinsicity of user click behavior to address this challenge.
  arXiv  Detail & Related papers  (2026-03-23T02:13:39Z)
• Beating the Winner's Curse via Inference-Aware Policy Optimization [26.01488014918074]
  A common approach is to train a machine learning model to predict counterfactual outcomes, and then select the policy that optimize the predicted objective value.<n>We propose a novel strategy called inference-aware policy optimization, which modifies policy optimization to account for how the policy will be evaluated downstream.
  arXiv  Detail & Related papers  (2025-10-20T23:28:12Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Conformal Off-Policy Evaluation in Markov Decision Processes [53.786439742572995]
  Reinforcement Learning aims at identifying and evaluating efficient control policies from data. Most methods for this learning task, referred to as Off-Policy Evaluation (OPE), do not come with accuracy and certainty guarantees. We present a novel OPE method based on Conformal Prediction that outputs an interval containing the true reward of the target policy with a prescribed level of certainty.
  arXiv  Detail & Related papers  (2023-04-05T16:45:11Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• SPEED: Experimental Design for Policy Evaluation in Linear Heteroscedastic Bandits [13.02672341061555]
  We study the problem of optimal data collection for policy evaluation in linear bandits. We first formulate an optimal design for weighted least squares estimates in the heteroscedastic linear bandit setting. We then use this formulation to derive the optimal allocation of samples per action during data collection.
  arXiv  Detail & Related papers  (2023-01-29T04:33:13Z)
• Counterfactual Learning with General Data-generating Policies [3.441021278275805]
  We develop an OPE method for a class of full support and deficient support logging policies in contextual-bandit settings. We prove that our method's prediction converges in probability to the true performance of a counterfactual policy as the sample size increases.
  arXiv  Detail & Related papers  (2022-12-04T21:07:46Z)
• Offline Policy Selection under Uncertainty [113.57441913299868]
  We consider offline policy selection as learning preferences over a set of policy prospects given a fixed experience dataset. Access to the full distribution over one's belief of the policy value enables more flexible selection algorithms under a wider range of downstream evaluation metrics. We show how BayesDICE may be used to rank policies with respect to any arbitrary downstream policy selection metric.
  arXiv  Detail & Related papers  (2020-12-12T23:09:21Z)
• Robust Batch Policy Learning in Markov Decision Processes [0.0]
  We study the offline data-driven sequential decision making problem in the framework of Markov decision process (MDP) We propose to evaluate each policy by a set of the average rewards with respect to distributions centered at the policy induced stationary distribution.
  arXiv  Detail & Related papers  (2020-11-09T04:41:21Z)
• Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
  In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy. We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
  arXiv  Detail & Related papers  (2020-11-08T23:16:19Z)
• Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies [80.42316902296832]
  We study the estimation of policy value and gradient of a deterministic policy from off-policy data when actions are continuous. In this setting, standard importance sampling and doubly robust estimators for policy value and gradient fail because the density ratio does not exist. We propose several new doubly robust estimators based on different kernelization approaches.
  arXiv  Detail & Related papers  (2020-06-06T15:52:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.