Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination

• URL: http://arxiv.org/abs/2405.18556v2
• Date: Mon, 3 Jun 2024 20:16:11 GMT
• Title: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
• Authors: Zhiyao Luo, Yangchen Pan, Peter Watkinson, Tingting Zhu,
• Abstract summary: offline reinforcement learning in dynamic treatment regimes presents a mix of unprecedented opportunities and challenges. We argue for a reassessment of applying RL in dynamic treatment regimes citing concerns such as inconsistent and potentially inconclusive evaluation metrics. We demonstrate that the performance of RL algorithms can significantly vary with changes in evaluation metrics and Markov Decision Process (MDP) formulations.
• Score: 7.162274565861427
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In the rapidly changing healthcare landscape, the implementation of offline reinforcement learning (RL) in dynamic treatment regimes (DTRs) presents a mix of unprecedented opportunities and challenges. This position paper offers a critical examination of the current status of offline RL in the context of DTRs. We argue for a reassessment of applying RL in DTRs, citing concerns such as inconsistent and potentially inconclusive evaluation metrics, the absence of naive and supervised learning baselines, and the diverse choice of RL formulation in existing research. Through a case study with more than 17,000 evaluation experiments using a publicly available Sepsis dataset, we demonstrate that the performance of RL algorithms can significantly vary with changes in evaluation metrics and Markov Decision Process (MDP) formulations. Surprisingly, it is observed that in some instances, RL algorithms can be surpassed by random baselines subjected to policy evaluation methods and reward design. This calls for more careful policy evaluation and algorithm development in future DTR works. Additionally, we discussed potential enhancements toward more reliable development of RL-based dynamic treatment regimes and invited further discussion within the community. Code is available at https://github.com/GilesLuo/ReassessDTR.

Related papers

• DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime [18.443316087890324]
  Reinforcement learning (RL) has garnered increasing recognition for its potential to optimise dynamic treatment regimes (DTRs) in personalised medicine. We introduce textitDTR-Bench, a benchmarking platform for simulating diverse healthcare scenarios. We evaluate various state-of-the-art RL algorithms across these settings, particularly highlighting their performance amidst real-world challenges.
  arXiv  Detail & Related papers  (2024-05-28T21:40:00Z)
• Timing Process Interventions with Causal Inference and Reinforcement Learning [2.919859121836811]
  This paper presents experiments on timed process interventions with synthetic data that renders genuine online RL and the comparison to CI possible. Our experiments reveal that RL's policies outperform those from CI and are more robust at the same time. Unlike CI, the unaltered online RL approach can be applied to other, more generic PresPM problems such as next best activity recommendations.
  arXiv  Detail & Related papers  (2023-06-07T10:02:16Z)
• A Survey on Causal Reinforcement Learning [41.645270300009436]
  We offer a review of Causal Reinforcement Learning (CRL) works, offer a review of CRL methods, and investigate the potential functionality from causality toward RL. In particular, we divide existing CRL approaches into two categories according to whether their causality-based information is given in advance or not. We analyze each category in terms of the formalization of different models, ranging from the Markov Decision Process (MDP), Partially Observed Markov Decision Process (POMDP), Multi-Arm Bandits (MAB), and Dynamic Treatment Regime (DTR)
  arXiv  Detail & Related papers  (2023-02-10T12:25:08Z)
• When does return-conditioned supervised learning work for offline reinforcement learning? [51.899892382786526]
  We study the capabilities and limitations of return-conditioned supervised learning. We find that RCSL returns the optimal policy under a set of assumptions stronger than those needed for the more traditional dynamic programming-based algorithms.
  arXiv  Detail & Related papers  (2022-06-02T15:05:42Z)
• Testing Stationarity and Change Point Detection in Reinforcement Learning [10.343546104340962]
  We develop a consistent procedure to test the nonstationarity of the optimal Q-function based on pre-collected historical data. We further develop a sequential change point detection method that can be naturally coupled with existing state-of-the-art RL methods for policy optimization in nonstationary environments.
  arXiv  Detail & Related papers  (2022-03-03T13:30:28Z)
• Pessimistic Model Selection for Offline Deep Reinforcement Learning [56.282483586473816]
  Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications. One main barrier is the over-fitting issue that leads to poor generalizability of the policy learned by DRL. We propose a pessimistic model selection (PMS) approach for offline DRL with a theoretical guarantee.
  arXiv  Detail & Related papers  (2021-11-29T06:29:49Z)
• On the Robustness of Controlled Deep Reinforcement Learning for Slice Placement [0.8459686722437155]
  We compare two Deep Reinforcement Learning algorithms: a pure DRL-based algorithm and a hybrid DRL as a hybrid DRL-heuristic algorithm. The evaluation results show that the proposed hybrid DRL-heuristic approach is more robust and reliable in case of unpredictable network load changes than pure DRL.
  arXiv  Detail & Related papers  (2021-08-05T10:24:33Z)
• Causal Inference Q-Network: Toward Resilient Reinforcement Learning [57.96312207429202]
  We consider a resilient DRL framework with observational interferences. Under this framework, we propose a causal inference based DRL algorithm called causal inference Q-network (CIQ) Our experimental results show that the proposed CIQ method could achieve higher performance and more resilience against observational interferences.
  arXiv  Detail & Related papers  (2021-02-18T23:50:20Z)
• Towards Continual Reinforcement Learning: A Review and Perspectives [69.48324517535549]
  We aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL) While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners. These include applications such as those in the fields of healthcare, education, logistics, and robotics.
  arXiv  Detail & Related papers  (2020-12-25T02:35:27Z)
• Critic Regularized Regression [70.8487887738354]
  We propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR) We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces.
  arXiv  Detail & Related papers  (2020-06-26T17:50:26Z)
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations [88.94162416324505]
  A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises. Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions. We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks.
  arXiv  Detail & Related papers  (2020-03-19T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.