Policy learning for many outcomes of interest: Combining optimal policy trees with multi-objective Bayesian optimisation

• URL: http://arxiv.org/abs/2212.06312v2
• Date: Tue, 17 Oct 2023 05:37:29 GMT
• Title: Policy learning for many outcomes of interest: Combining optimal policy trees with multi-objective Bayesian optimisation
• Authors: Patrick Rehill and Nicholas Biddle
• Abstract summary: Multi-Objective Policy Learning combines optimal decision trees for policy learning with a multi-objective Bayesian optimisation approach. The method is applied to a real-world case-study of non-price rationing of anti-malarial medication in Kenya.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Methods for learning optimal policies use causal machine learning models to create human-interpretable rules for making choices around the allocation of different policy interventions. However, in realistic policy-making contexts, decision-makers often care about trade-offs between outcomes, not just single-mindedly maximising utility for one outcome. This paper proposes an approach termed Multi-Objective Policy Learning (MOPoL) which combines optimal decision trees for policy learning with a multi-objective Bayesian optimisation approach to explore the trade-off between multiple outcomes. It does this by building a Pareto frontier of non-dominated models for different hyperparameter settings which govern outcome weighting. The key here is that a low-cost greedy tree can be an accurate proxy for the very computationally costly optimal tree for the purposes of making decisions which means models can be repeatedly fit to learn a Pareto frontier. The method is applied to a real-world case-study of non-price rationing of anti-malarial medication in Kenya.

Related papers

• Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning [14.260168974085376]
  This paper investigates multi-objective reinforcement learning (MORL) It focuses on learning optimal policies in the presence of multiple reward functions. Despite MORL's success, there is still a lack of satisfactory understanding of various MORL optimization targets and efficient learning algorithms.
  arXiv  Detail & Related papers  (2024-07-24T17:58:49Z)
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients [62.81324245896716]
  Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. In common practice, convergence (hyper)policies are learned only to deploy their deterministic version. We show how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy.
  arXiv  Detail & Related papers  (2024-05-03T16:45:15Z)
• UCB-driven Utility Function Search for Multi-objective Reinforcement Learning [75.11267478778295]
  In Multi-objective Reinforcement Learning (MORL) agents are tasked with optimising decision-making behaviours. We focus on the case of linear utility functions parameterised by weight vectors w. We introduce a method based on Upper Confidence Bound to efficiently search for the most promising weight vectors during different stages of the learning process.
  arXiv  Detail & Related papers  (2024-05-01T09:34:42Z)
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data [102.16105233826917]
  Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning.
  arXiv  Detail & Related papers  (2024-04-22T17:20:18Z)
• Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning [13.627087954965695]
  We propose a human-in-the-loop policy optimization framework for preference-based MORL. Our method proactively learns the DM's implicit preference information without requiring any priori knowledge. We evaluate our approach against three conventional MORL algorithms and four state-of-the-art preference-based MORL algorithms.
  arXiv  Detail & Related papers  (2024-01-04T09:17:53Z)
• Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism [91.52263068880484]
  We study offline Reinforcement Learning with Human Feedback (RLHF) We aim to learn the human's underlying reward and the MDP's optimal policy from a set of trajectories induced by human choices. RLHF is challenging for multiple reasons: large state space but limited human feedback, the bounded rationality of human decisions, and the off-policy distribution shift.
  arXiv  Detail & Related papers  (2023-05-29T01:18:39Z)
• Optimal Decision Tree Policies for Markov Decision Processes [7.995360025953931]
  We study the optimization of size-limited decision trees for Markov Decision Processes (MPDs) We show that this is due to an inherent shortcoming of imitation learning, namely that complex policies cannot be represented using size-limited trees. While there is generally a trade-off between the performance and interpretability of machine learning models, we find that OMDTs limited to a depth of 3 often perform close to the optimal limit.
  arXiv  Detail & Related papers  (2023-01-30T18:51:02Z)
• Pareto Set Learning for Expensive Multi-Objective Optimization [5.419608513284392]
  Expensive multi-objective optimization problems can be found in many real-world applications. This paper develops a novel learning-based method to approximate the whole Pareto set for MOBO.
  arXiv  Detail & Related papers  (2022-10-16T09:41:54Z)
• Off-Policy Evaluation with Policy-Dependent Optimization Response [90.28758112893054]
  We develop a new framework for off-policy evaluation with a textitpolicy-dependent linear optimization response. We construct unbiased estimators for the policy-dependent estimand by a perturbation method. We provide a general algorithm for optimizing causal interventions.
  arXiv  Detail & Related papers  (2022-02-25T20:25:37Z)
• An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes [0.0]
  Exploitation vs Caution (EvC) is a paradigm that elegantly incorporates model uncertainty abiding by the Bayesian formalism. We validate EvC with state-of-the-art approaches in different discrete, yet simple, environments offering a fair variety of MDP classes. In the tested scenarios EvC manages to select robust policies and hence stands out as a useful tool for practitioners.
  arXiv  Detail & Related papers  (2021-05-27T20:12:20Z)
• Generalized and Scalable Optimal Sparse Decision Trees [56.35541305670828]
  We present techniques that produce optimal decision trees over a variety of objectives. We also introduce a scalable algorithm that produces provably optimal results in the presence of continuous variables.
  arXiv  Detail & Related papers  (2020-06-15T19:00:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.