Near-Equivalent Q-learning Policies for Dynamic Treatment Regimes

• URL: http://arxiv.org/abs/2603.19440v1
• Date: Thu, 19 Mar 2026 20:08:27 GMT
• Title: Near-Equivalent Q-learning Policies for Dynamic Treatment Regimes
• Abstract summary: We extend the Q-learning framework for retrospective data by introducing a worst-value tolerance criterion.<n>We show that the proposed approach constructs sets of $varepsilon$-optimal policies whose performance remains within a controlled neighborhood of the optimum.<n>We illustrate the framework in two settings: a single-stage problem highlighting indifference regions around the decision boundary, and a multi-stage decision process based on a simulated oncology model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Precision medicine aims to tailor therapeutic decisions to individual patient characteristics. This objective is commonly formalized through dynamic treatment regimes, which use statistical and machine learning methods to derive sequential decision rules adapted to evolving clinical information. In most existing formulations, these approaches produce a single optimal treatment at each stage, leading to a unique decision sequence. However, in many clinical settings, several treatment options may yield similar expected outcomes, and focusing on a single optimal policy may conceal meaningful alternatives. We extend the Q-learning framework for retrospective data by introducing a worst-value tolerance criterion controlled by a hyperparameter $\varepsilon$, which specifies the maximum acceptable deviation from the optimal expected value. Rather than identifying a single optimal policy, the proposed approach constructs sets of $\varepsilon$-optimal policies whose performance remains within a controlled neighborhood of the optimum. This formulation shifts Q-learning from a vector-valued representation to a matrix-valued one, allowing multiple admissible value functions to coexist during backward recursion. The approach yields families of near-equivalent treatment strategies and explicitly identifies regions of treatment indifference where several decisions achieve comparable outcomes. We illustrate the framework in two settings: a single-stage problem highlighting indifference regions around the decision boundary, and a multi-stage decision process based on a simulated oncology model describing tumor size and treatment toxicity dynamics.

Related papers

• Set-Valued Policy Learning [17.33185980499072]
  We propose a set-valued policy learning paradigm for the multiple-treatment setting, in which policies output a set of plausible treatments rather than a single recommendation.<n>We extend the learning-to-defer framework to multiple treatments via a novel textitgreatest Lower Bound method, and introduce textitconformal policy learning, which bridges the gap between unobserved ground-truth optimal treatments and estimated optimal treatment rules.
  arXiv  Detail & Related papers  (2026-05-19T13:24:26Z)
• On Multiple Robustness of Proximal Dynamic Treatment Regimes [6.4049477424590355]
  Estimating optimal dynamic treatment regimes via sequential randomized trials might face costly and ethical hurdles.<n>We propose proximal causal inference framework for learning optimal dynamic treatment regimes when the unconfoundedness assumption fails.
  arXiv  Detail & Related papers  (2025-10-23T11:38:40Z)
• An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
  We first construct a max-margin optimization-based model to model potentially non-monotonic preferences. We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration. Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
  arXiv  Detail & Related papers  (2024-09-04T14:36:20Z)
• Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs [82.34567890576423]
  We develop a deterministic policy gradient primal-dual method to find an optimal deterministic policy with non-asymptotic convergence.<n>We prove that the primal-dual iterates of D-PGPD converge at a sub-linear rate to an optimal regularized primal-dual pair.<n>This appears to be the first work that proposes a deterministic policy search method for continuous-space constrained MDPs.
  arXiv  Detail & Related papers  (2024-08-19T14:11:04Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Stage-Aware Learning for Dynamic Treatments [3.6923632650826486]
  We propose a novel individualized learning method for dynamic treatment regimes. By relaxing the restriction that the observed trajectory must be fully aligned with the optimal treatments, our approach substantially improves the sample efficiency and stability of IPWE-based methods.
  arXiv  Detail & Related papers  (2023-10-30T06:35:31Z)
• A Flexible Framework for Incorporating Patient Preferences Into Q-Learning [1.2999518604217852]
  We propose a latent model approach that adapts Q-learning to tackle the aforementioned difficulties.<n>We conduct simulation experiments based on an ongoing trial for low back pain as well as a well-known trial for schizophrenia.<n>LUQ-Learning achieves highly competitive performance compared to alternative baselines.
  arXiv  Detail & Related papers  (2023-07-22T08:58:07Z)
• High-probability sample complexities for policy evaluation with linear function approximation [88.87036653258977]
  We investigate the sample complexities required to guarantee a predefined estimation error of the best linear coefficients for two widely-used policy evaluation algorithms. We establish the first sample complexity bound with high-probability convergence guarantee that attains the optimal dependence on the tolerance level.
  arXiv  Detail & Related papers  (2023-05-30T12:58:39Z)
• Asymptotic Inference for Multi-Stage Stationary Treatment Policy with Variable Selection [13.202945240520986]
  Dynamic treatment regimes or policies are a sequence of decision functions over multiple stages that are tailored to individual features.<n>One important class of treatment policies in practice, namely multi-stage stationary treatment policies, prescribes treatment assignment using the same decision function across stages.<n>Although there has been extensive literature on constructing valid inference for the value function associated with dynamic treatment policies, little work has focused on the policies themselves.
  arXiv  Detail & Related papers  (2023-01-29T22:00:53Z)
• Off-Policy Evaluation with Policy-Dependent Optimization Response [90.28758112893054]
  We develop a new framework for off-policy evaluation with a textitpolicy-dependent linear optimization response. We construct unbiased estimators for the policy-dependent estimand by a perturbation method. We provide a general algorithm for optimizing causal interventions.
  arXiv  Detail & Related papers  (2022-02-25T20:25:37Z)
• Optimal discharge of patients from intensive care via a data-driven policy learning framework [58.720142291102135]
  It is important that the patient discharge task addresses the nuanced trade-off between decreasing a patient's length of stay and the risk of readmission or even death following the discharge decision. This work introduces an end-to-end general framework for capturing this trade-off to recommend optimal discharge timing decisions. A data-driven approach is used to derive a parsimonious, discrete state space representation that captures a patient's physiological condition.
  arXiv  Detail & Related papers  (2021-12-17T04:39:33Z)
• Estimation of Optimal Dynamic Treatment Assignment Rules under Policy Constraints [0.0]
  We study estimation of an optimal dynamic treatment regime that guides the optimal treatment assignment for each individual at each stage based on their history. The paper proposes two estimation methods: one solves the treatment assignment problem sequentially through backward induction, and the other solves the entire problem simultaneously across all stages.
  arXiv  Detail & Related papers  (2021-06-09T12:42:53Z)
• Multicategory Angle-based Learning for Estimating Optimal Dynamic Treatment Regimes with Censored Data [12.499787110182632]
  An optimal treatment regime (DTR) consists of a sequence of decision rules in maximizing long-term benefits. In this paper, we develop a novel angle-based approach to target the optimal DTR under a multicategory treatment framework. Our numerical studies show that the proposed method outperforms competing methods in terms of maximizing the conditional survival function.
  arXiv  Detail & Related papers  (2020-01-14T05:19:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.