Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning

• URL: http://arxiv.org/abs/2412.05783v1
• Date: Sun, 08 Dec 2024 02:28:58 GMT
• Title: Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning
• Authors: Shuguang Yu, Shuxing Fang, Ruixin Peng, Zhengling Qi, Fan Zhou, Chengchun Shi,
• Abstract summary: We propose a two-way unmeasured confounding assumption to model the system dynamics in causal reinforcement learning. We develop a two-way deconfounder algorithm that devises a neural tensor network to simultaneously learn both the unmeasured confounders and the system dynamics.
• Score: 17.49812746668935
• License:
• Abstract: This paper studies off-policy evaluation (OPE) in the presence of unmeasured confounders. Inspired by the two-way fixed effects regression model widely used in the panel data literature, we propose a two-way unmeasured confounding assumption to model the system dynamics in causal reinforcement learning and develop a two-way deconfounder algorithm that devises a neural tensor network to simultaneously learn both the unmeasured confounders and the system dynamics, based on which a model-based estimator can be constructed for consistent policy value estimation. We illustrate the effectiveness of the proposed estimator through theoretical results and numerical experiments.

Related papers

• Imitation Learning from Observations: An Autoregressive Mixture of Experts Approach [2.4427666827706074]
  This paper presents a novel approach to imitation learning from observations, where an autoregressive mixture of experts model is deployed to fit the underlying policy. The effectiveness of the proposed framework is validated using two autonomous driving datasets collected from human demonstrations.
  arXiv  Detail & Related papers  (2024-11-12T22:56:28Z)
• Self-Improving Interference Management Based on Deep Learning With Uncertainty Quantification [10.403513606082067]
  This paper presents a self-improving interference management framework tailored for wireless communications. Our approach addresses the computational challenges inherent in traditional optimization-based algorithms. A breakthrough of our framework is its acknowledgment of the limitations inherent in data-driven models.
  arXiv  Detail & Related papers  (2024-01-24T03:28:48Z)
• Enhancing Scalability in Recommender Systems through Lottery Ticket Hypothesis and Knowledge Distillation-based Neural Network Pruning [1.3654846342364308]
  This study introduces an innovative approach aimed at the efficient pruning of neural networks, with a particular focus on their deployment on edge devices. Our method involves the integration of the Lottery Ticket Hypothesis (LTH) with the Knowledge Distillation (KD) framework, resulting in the formulation of three distinct pruning models. Gratifyingly, our approaches yielded a GPU computation-power reduction of up to 66.67%.
  arXiv  Detail & Related papers  (2024-01-19T04:17:50Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)
• Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study [71.84852429039881]
  Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise. We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
  arXiv  Detail & Related papers  (2023-06-13T12:43:59Z)
• Off-policy evaluation for learning-to-rank via interpolating the item-position model and the position-based model [83.83064559894989]
  A critical need for industrial recommender systems is the ability to evaluate recommendation policies offline, before deploying them to production. We develop a new estimator that mitigates the problems of the two most popular off-policy estimators for rankings. In particular, the new estimator, called INTERPOL, addresses the bias of a potentially misspecified position-based model.
  arXiv  Detail & Related papers  (2022-10-15T17:22:30Z)
• Discriminator Augmented Model-Based Reinforcement Learning [47.094522301093775]
  It is common in practice for the learned model to be inaccurate, impairing planning and leading to poor performance. This paper aims to improve planning with an importance sampling framework that accounts for discrepancy between the true and learned dynamics.
  arXiv  Detail & Related papers  (2021-03-24T06:01:55Z)
• Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction [37.06232589005015]
  Value function is the central notion of Reinforcement Learning (RL) We propose Value Decomposition with Future Prediction (VDFP) We analytically decompose the value function into a latent future dynamics part and a policy-independent trajectory return part, inducing a way to model latent dynamics and returns separately in value estimation.
  arXiv  Detail & Related papers  (2021-03-03T07:28:56Z)
• On the model-based stochastic value gradient for continuous reinforcement learning [50.085645237597056]
  We show that simple model-based agents can outperform state-of-the-art model-free agents in terms of both sample-efficiency and final reward. Our findings suggest that model-based policy evaluation deserves closer attention.
  arXiv  Detail & Related papers  (2020-08-28T17:58:29Z)
• Control as Hybrid Inference [62.997667081978825]
  We present an implementation of CHI which naturally mediates the balance between iterative and amortised inference. We verify the scalability of our algorithm on a continuous control benchmark, demonstrating that it outperforms strong model-free and model-based baselines.
  arXiv  Detail & Related papers  (2020-07-11T19:44:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.