An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes

• URL: http://arxiv.org/abs/2509.26429v1
• Date: Tue, 30 Sep 2025 15:49:29 GMT
• Title: An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes
• Authors: Emil Javurek, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Dennis Frauen, Stefan Feuerriegel,
• Abstract summary: We develop a novel meta-learner called DRQ-learner.<n>Our DRQ-learner is applicable to settings with both discrete and continuous state spaces.
• Score: 55.93922317950527
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predicting individualized potential outcomes in sequential decision-making is central for optimizing therapeutic decisions in personalized medicine (e.g., which dosing sequence to give to a cancer patient). However, predicting potential outcomes over long horizons is notoriously difficult. Existing methods that break the curse of the horizon typically lack strong theoretical guarantees such as orthogonality and quasi-oracle efficiency. In this paper, we revisit the problem of predicting individualized potential outcomes in sequential decision-making (i.e., estimating Q-functions in Markov decision processes with observational data) through a causal inference lens. In particular, we develop a comprehensive theoretical foundation for meta-learners in this setting with a focus on beneficial theoretical properties. As a result, we yield a novel meta-learner called DRQ-learner and establish that it is: (1) doubly robust (i.e., valid inference under the misspecification of one of the nuisances), (2) Neyman-orthogonal (i.e., insensitive to first-order estimation errors in the nuisance functions), and (3) achieves quasi-oracle efficiency (i.e., behaves asymptotically as if the ground-truth nuisance functions were known). Our DRQ-learner is applicable to settings with both discrete and continuous state spaces. Further, our DRQ-learner is flexible and can be used together with arbitrary machine learning models (e.g., neural networks). We validate our theoretical results through numerical experiments, thereby showing that our meta-learner outperforms state-of-the-art baselines.

Related papers

• Is Uncertainty Quantification a Viable Alternative to Learned Deferral? [0.5156484100374058]
  One aspect of AI safety is the models' ability to defer decisions to a human expert.<n>During clinical translation, models often face challenges such as data shift.<n>Uncertainty quantification methods may be a promising choice for AI deferral.
  arXiv  Detail & Related papers  (2025-08-04T11:37:59Z)
• A Relative Ignorability Framework for Decision-Relevant Observability in Control Theory and Reinforcement Learning [0.0]
  We show Causal inference paradigms formalise ignorability of missingness.<n>Relative ignorability is a graphical-causal criterion which refines the requirements for accurate decision-making.
  arXiv  Detail & Related papers  (2025-04-10T13:15:52Z)
• Inadequacy of common stochastic neural networks for reliable clinical decision support [0.4262974002462632]
  Widespread adoption of AI for medical decision making is still hindered due to ethical and safety-related concerns. Common deep learning approaches, however, have the tendency towards overconfidence under data shift. This study investigates their actual reliability in clinical applications.
  arXiv  Detail & Related papers  (2024-01-24T18:49:30Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)
• Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders [16.193776814471768]
  We study robust policy evaluation and policy optimization in the presence of sequentially-exogenous unobserved confounders. We provide sample complexity bounds, insights, and show effectiveness both in simulations and on real-world longitudinal healthcare data of treating sepsis.
  arXiv  Detail & Related papers  (2023-02-01T18:40:53Z)
• Deep Learning Methods for Proximal Inference via Maximum Moment Restriction [0.0]
  We introduce a flexible and scalable method based on a deep neural network to estimate causal effects in the presence of unmeasured confounding. Our method achieves state of the art performance on two well-established proximal inference benchmarks.
  arXiv  Detail & Related papers  (2022-05-19T19:51:42Z)
• On the Benefits of Large Learning Rates for Kernel Methods [110.03020563291788]
  We show that a phenomenon can be precisely characterized in the context of kernel methods. We consider the minimization of a quadratic objective in a separable Hilbert space, and show that with early stopping, the choice of learning rate influences the spectral decomposition of the obtained solution.
  arXiv  Detail & Related papers  (2022-02-28T13:01:04Z)
• Masked prediction tasks: a parameter identifiability view [49.533046139235466]
  We focus on the widely used self-supervised learning method of predicting masked tokens. We show that there is a rich landscape of possibilities, out of which some prediction tasks yield identifiability, while others do not.
  arXiv  Detail & Related papers  (2022-02-18T17:09:32Z)
• The Causal Neural Connection: Expressiveness, Learnability, and Inference [125.57815987218756]
  An object called structural causal model (SCM) represents a collection of mechanisms and sources of random variation of the system under investigation. In this paper, we show that the causal hierarchy theorem (Thm. 1, Bareinboim et al., 2020) still holds for neural models. We introduce a special type of SCM called a neural causal model (NCM), and formalize a new type of inductive bias to encode structural constraints necessary for performing causal inferences.
  arXiv  Detail & Related papers  (2021-07-02T01:55:18Z)
• General stochastic separation theorems with optimal bounds [68.8204255655161]
  Phenomenon of separability was revealed and used in machine learning to correct errors of Artificial Intelligence (AI) systems and analyze AI instabilities. Errors or clusters of errors can be separated from the rest of the data. The ability to correct an AI system also opens up the possibility of an attack on it, and the high dimensionality induces vulnerabilities caused by the same separability.
  arXiv  Detail & Related papers  (2020-10-11T13:12:41Z)
• Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint [35.5156045701898]
  We provide a novel viewpoint of regularization-based continual learning by formulating it as a second-order Taylor approximation of the loss function of each task. Based on this viewpoint, we study the optimization aspects (i.e., convergence) as well as generalization properties (i.e., finite-sample guarantees) of regularization-based continual learning.
  arXiv  Detail & Related papers  (2020-06-19T06:08:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.