IntOPE: Off-Policy Evaluation in the Presence of Interference

• URL: http://arxiv.org/abs/2408.13484v1
• Date: Sat, 24 Aug 2024 06:07:25 GMT
• Title: IntOPE: Off-Policy Evaluation in the Presence of Interference
• Authors: Yuqi Bai, Ziyu Zhao, Minqin Zhu, Kun Kuang,
• Abstract summary: Off-Policy Evaluation (OPE) is employed to assess the potential impact of a hypothetical policy. IntIPW is an IPW-style estimator that integrates marginalized importance weights to account for both individual actions and the influence of adjacent entities.
• Score: 23.167697252901398
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Off-Policy Evaluation (OPE) is employed to assess the potential impact of a hypothetical policy using logged contextual bandit feedback, which is crucial in areas such as personalized medicine and recommender systems, where online interactions are associated with significant risks and costs. Traditionally, OPE methods rely on the Stable Unit Treatment Value Assumption (SUTVA), which assumes that the reward for any given individual is unaffected by the actions of others. However, this assumption often fails in real-world scenarios due to the presence of interference, where an individual's reward is affected not just by their own actions but also by the actions of their peers. This realization reveals significant limitations of existing OPE methods in real-world applications. To address this limitation, we propose IntIPW, an IPW-style estimator that extends the Inverse Probability Weighting (IPW) framework by integrating marginalized importance weights to account for both individual actions and the influence of adjacent entities. Extensive experiments are conducted on both synthetic and real-world data to demonstrate the effectiveness of the proposed IntIPW method.

Related papers

• Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences [24.361550505778155]
  Offcommerce evaluation (OPE) is widely applied in sectors such as pharmaceuticals and e-policy-policy. This paper introduces a causal deepset framework that relaxes several key structural assumptions. We present novel algorithms that incorporate the PI assumption into OPE and thoroughly examine their theoretical foundations.
  arXiv  Detail & Related papers  (2024-07-25T10:02:11Z)
• Individualized Policy Evaluation and Learning under Clustered Network Interference [4.560284382063488]
  We consider the problem of evaluating and learning an optimal individualized treatment rule under clustered network interference. We propose an estimator that can be used to evaluate the empirical performance of an ITR. We derive the finite-sample regret bound for a learned ITR, showing that the use of our efficient evaluation estimator leads to the improved performance of learned policies.
  arXiv  Detail & Related papers  (2023-11-04T17:58:24Z)
• Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation [137.3520153445413]
  A notable gap exists in the evaluation of causal discovery methods, where insufficient emphasis is placed on downstream inference. We evaluate seven established baseline causal discovery methods including a newly proposed method based on GFlowNets. The results of our study demonstrate that some of the algorithms studied are able to effectively capture a wide range of useful and diverse ATE modes.
  arXiv  Detail & Related papers  (2023-07-11T02:58:10Z)
• Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling [30.835774920236872]
  We study off-policy evaluation of contextual bandit policies for large discrete action spaces. We propose a new estimator, called OffCEM, that is based on the conjunct effect model (CEM), a novel decomposition of the causal effect into a cluster effect and a residual effect. Experiments demonstrate that OffCEM provides substantial improvements in OPE especially in the presence of many actions.
  arXiv  Detail & Related papers  (2023-05-14T04:16:40Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• Hallucinated Adversarial Control for Conservative Offline Policy Evaluation [64.94009515033984]
  We study the problem of conservative off-policy evaluation (COPE) where given an offline dataset of environment interactions, we seek to obtain a (tight) lower bound on a policy's performance. We introduce HAMBO, which builds on an uncertainty-aware learned model of the transition dynamics. We prove that the resulting COPE estimates are valid lower bounds, and, under regularity conditions, show their convergence to the true expected return.
  arXiv  Detail & Related papers  (2023-03-02T08:57:35Z)
• Debiasing Recommendation by Learning Identifiable Latent Confounders [49.16119112336605]
  Confounding bias arises due to the presence of unmeasured variables that can affect both a user's exposure and feedback. Existing methods either (1) make untenable assumptions about these unmeasured variables or (2) directly infer latent confounders from users' exposure. We propose a novel method, i.e., identifiable deconfounder (iDCF), which leverages a set of proxy variables to resolve the aforementioned non-identification issue.
  arXiv  Detail & Related papers  (2023-02-10T05:10:26Z)
• An Instrumental Variable Approach to Confounded Off-Policy Evaluation [11.785128674216903]
  Off-policy evaluation (OPE) is a method for estimating the return of a target policy. This paper develops an instrumental variable (IV)-based method for consistent OPE in confounded Markov decision processes.
  arXiv  Detail & Related papers  (2022-12-29T22:06:51Z)
• Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process [14.828039846764549]
  We show that with some auxiliary variables that mediate the effect of actions on the system dynamics, the target policy's value is identifiable in a confounded Markov decision process. Our method is justified by theoretical results, simulated and real datasets obtained from ridesharing companies.
  arXiv  Detail & Related papers  (2022-02-22T00:03:48Z)
• Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes [65.91730154730905]
  In applications of offline reinforcement learning to observational data, such as in healthcare or education, a general concern is that observed actions might be affected by unobserved factors. Here we tackle this by considering off-policy evaluation in a partially observed Markov decision process (POMDP) We extend the framework of proximal causal inference to our POMDP setting, providing a variety of settings where identification is made possible.
  arXiv  Detail & Related papers  (2021-10-28T17:46:14Z)
• Learning "What-if" Explanations for Sequential Decision-Making [92.8311073739295]
  Building interpretable parameterizations of real-world decision-making on the basis of demonstrated behavior is essential. We propose learning explanations of expert decisions by modeling their reward function in terms of preferences with respect to "what if" outcomes. We highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
  arXiv  Detail & Related papers  (2020-07-02T14:24:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.