Pragmatic Policy Development via Interpretable Behavior Cloning

• URL: http://arxiv.org/abs/2507.17056v1
• Date: Tue, 22 Jul 2025 22:34:35 GMT
• Title: Pragmatic Policy Development via Interpretable Behavior Cloning
• Authors: Anton Matsson, Yaochen Rao, Heather J. Litman, Fredrik D. Johansson,
• Abstract summary: We propose deriving treatment policies from the most frequently chosen actions in each patient state, as estimated by an interpretable model of the behavior policy.<n>We demonstrate that policies derived under this framework can outperform current practice, offering interpretable alternatives to those obtained via offline RL.
• Score: 6.177449809243359
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline reinforcement learning (RL) holds great promise for deriving optimal policies from observational data, but challenges related to interpretability and evaluation limit its practical use in safety-critical domains. Interpretability is hindered by the black-box nature of unconstrained RL policies, while evaluation -- typically performed off-policy -- is sensitive to large deviations from the data-collecting behavior policy, especially when using methods based on importance sampling. To address these challenges, we propose a simple yet practical alternative: deriving treatment policies from the most frequently chosen actions in each patient state, as estimated by an interpretable model of the behavior policy. By using a tree-based model, which is specifically designed to exploit patterns in the data, we obtain a natural grouping of states with respect to treatment. The tree structure ensures interpretability by design, while varying the number of actions considered controls the degree of overlap with the behavior policy, enabling reliable off-policy evaluation. This pragmatic approach to policy development standardizes frequent treatment patterns, capturing the collective clinical judgment embedded in the data. Using real-world examples in rheumatoid arthritis and sepsis care, we demonstrate that policies derived under this framework can outperform current practice, offering interpretable alternatives to those obtained via offline RL.

Related papers

• Distributionally Robust Policy Evaluation and Learning for Continuous Treatment with Observational Data [20.125760566665704]
  Using offline observational data for policy evaluation and learning allows decision-makers to evaluate and learn a policy that connects characteristics and interventions.<n>Most existing literature has focused on either discrete treatment spaces or assumed no difference in the distributions between the policy-learning and policy-deployed environments.<n>This paper focuses on developing a distributionally robust policy under a continuous treatment setting.
  arXiv  Detail & Related papers  (2025-01-18T08:12:56Z)
• Optimizing Warfarin Dosing Using Contextual Bandit: An Offline Policy Learning and Evaluation Method [2.8806234438838256]
  Warfarin, an anticoagulant medication, is formulated to prevent and address conditions associated with abnormal blood clotting. Finding the suitable dosage remains challenging due to individual response variations, and prescribing an incorrect dosage may lead to severe consequences. We used contextual bandit and reinforcement learning to determine the optimal personalized dosage strategy.
  arXiv  Detail & Related papers  (2024-02-16T23:13:05Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)
• Offline Reinforcement Learning with On-Policy Q-Function Regularization [57.09073809901382]
  We deal with the (potentially catastrophic) extrapolation error induced by the distribution shift between the history dataset and the desired policy. We propose two algorithms taking advantage of the estimated Q-function through regularizations, and demonstrate they exhibit strong performance on the D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-07-25T21:38:08Z)
• Hallucinated Adversarial Control for Conservative Offline Policy Evaluation [64.94009515033984]
  We study the problem of conservative off-policy evaluation (COPE) where given an offline dataset of environment interactions, we seek to obtain a (tight) lower bound on a policy's performance. We introduce HAMBO, which builds on an uncertainty-aware learned model of the transition dynamics. We prove that the resulting COPE estimates are valid lower bounds, and, under regularity conditions, show their convergence to the true expected return.
  arXiv  Detail & Related papers  (2023-03-02T08:57:35Z)
• Offline Policy Evaluation and Optimization under Confounding [35.778917456294046]
  We map out the landscape of offline policy evaluation for confounded MDPs. We characterize settings where consistent value estimates are provably not achievable. We present new algorithms for offline policy improvement and prove local convergence guarantees.
  arXiv  Detail & Related papers  (2022-11-29T20:45:08Z)
• Offline Reinforcement Learning with Soft Behavior Regularization [0.8937096931077437]
  In this work, we derive a new policy learning objective that can be used in the offline setting. Unlike state-independent regularization used in prior approaches, this textitsoft regularization allows more freedom of policy deviation. Our experimental results show that SBAC matches or outperforms the state-of-the-art on a set of continuous control locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2021-10-14T14:29:44Z)
• Continuous Action Reinforcement Learning from a Mixture of Interpretable Experts [35.80418547105711]
  We propose a policy scheme that retains a complex function approxor for its internal value predictions but constrains the policy to have a concise, hierarchical, and human-readable structure. The main technical contribution of the paper is to address the challenges introduced by this non-differentiable state selection procedure.
  arXiv  Detail & Related papers  (2020-06-10T16:02:08Z)
• Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies [80.42316902296832]
  We study the estimation of policy value and gradient of a deterministic policy from off-policy data when actions are continuous. In this setting, standard importance sampling and doubly robust estimators for policy value and gradient fail because the density ratio does not exist. We propose several new doubly robust estimators based on different kernelization approaches.
  arXiv  Detail & Related papers  (2020-06-06T15:52:05Z)
• Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning [80.42316902296832]
  We study the efficient off-policy evaluation of natural policies, which are defined in terms of deviations from the behavior policy. This is a departure from the literature on off-policy evaluation where most work consider the evaluation of explicitly specified policies.
  arXiv  Detail & Related papers  (2020-06-06T15:08:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.