Multi-Agent Reinforcement Learning for Intraday Operating Rooms Scheduling under Uncertainty

• URL: http://arxiv.org/abs/2512.04918v1
• Date: Thu, 04 Dec 2025 15:47:08 GMT
• Title: Multi-Agent Reinforcement Learning for Intraday Operating Rooms Scheduling under Uncertainty
• Authors: Kailiang Liu, Ying Chen, Ralf Borndörfer, Thorsten Koch,
• Abstract summary: Intraday surgical scheduling is a multi-objective decision problem under uncertainty-balancing throughput, urgent and emergency demand, delays, sequence-dependent setups, and overtime.<n>We formulate the problem as a cooperative Markov game and propose a multi-agent reinforcement learning framework in which each operating room is an agent trained with centralized training and decentralized execution.<n>All agents share a policy trained via Proximal Policy Optimization (PPO), which maps rich system states to actions, while a within-epoch sequential assignment protocol constructs joint schedules across ORs.
• Score: 4.5515292789901975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Intraday surgical scheduling is a multi-objective decision problem under uncertainty-balancing elective throughput, urgent and emergency demand, delays, sequence-dependent setups, and overtime. We formulate the problem as a cooperative Markov game and propose a multi-agent reinforcement learning (MARL) framework in which each operating room (OR) is an agent trained with centralized training and decentralized execution. All agents share a policy trained via Proximal Policy Optimization (PPO), which maps rich system states to actions, while a within-epoch sequential assignment protocol constructs conflict-free joint schedules across ORs. A mixed-integer pre-schedule provides reference starting times for electives; we impose type-specific quadratic delay penalties relative to these references and a terminal overtime penalty, yielding a single reward that captures throughput, timeliness, and staff workload. In simulations reflecting a realistic hospital mix (six ORs, eight surgery types, random urgent and emergency arrivals), the learned policy outperforms six rule-based heuristics across seven metrics and three evaluation subsets, and, relative to an ex post MIP oracle, quantifies optimality gaps. Policy analytics reveal interpretable behavior-prioritizing emergencies, batching similar cases to reduce setups, and deferring lower-value electives. We also derive a suboptimality bound for the sequential decomposition under simplifying assumptions. We discuss limitations-including OR homogeneity and the omission of explicit staffing constraints-and outline extensions. Overall, the approach offers a practical, interpretable, and tunable data-driven complement to optimization for real-time OR scheduling.

Related papers

• Coverage Improvement and Fast Convergence of On-policy Preference Learning [67.36750525893514]
  Online on-policy preference learning algorithms for language model alignment can significantly outperform their offline counterparts.<n>We analyze how the sampling policy's coverage evolves throughout on-policy training.<n>We develop principled on-policy schemes for reward distillation in the general function class setting.
  arXiv  Detail & Related papers  (2026-01-13T10:46:06Z)
• Multi-Action Self-Improvement for Neural Combinatorial Optimization [0.979731979071071]
  Self-improvement models iteratively refine their policies by generating and imitating high-quality solutions.<n>These approaches fail to exploit the structure of problems involving the coordination of multiple agents.<n>We extend self-improvement to operate over joint multi-agent actions.
  arXiv  Detail & Related papers  (2025-10-14T08:26:27Z)
• Reinforcement learning with combinatorial actions for coupled restless bandits [62.89013331120493]
  We propose SEQUOIA, an RL algorithm that directly optimize for long-term reward over the feasible action space.<n>We empirically validate SEQUOIA on four novel restless bandit problems with constraints: multiple interventions, path constraints, bipartite matching, and capacity constraints.
  arXiv  Detail & Related papers  (2025-03-01T21:25:21Z)
• Autoregressive Policy Optimization for Constrained Allocation Tasks [4.316765170255551]
  We propose a new method for constrained allocation tasks based on an autoregressive process to sequentially sample allocations for each entity. In addition, we introduce a novel de-biasing mechanism to counter the initial bias caused by sequential sampling.
  arXiv  Detail & Related papers  (2024-09-27T13:27:15Z)
• Towards Fast Rates for Federated and Multi-Task Reinforcement Learning [34.34798425737858]
  We propose Fast-FedPG, a novel federated policy algorithm with a carefully designed bias-correction mechanism. Under a gradient-domination condition, we prove that our algorithm guarantees (i) fast linear convergence with exact gradients, and (ii) sub-linear rates that enjoy a linear speedup w.r.t. the number of agents with noisy, truncated policy gradients.
  arXiv  Detail & Related papers  (2024-09-09T02:59:17Z)
• Generalization Bounds of Surrogate Policies for Combinatorial Optimization Problems [53.03951222945921]
  We analyze smoothed (perturbed) policies, adding controlled random perturbations to the direction used by the linear oracle.<n>Our main contribution is a generalization bound that decomposes the excess risk into perturbation bias, statistical estimation error, and optimization error.<n>We illustrate the scope of the results on applications such as vehicle scheduling, highlighting how smoothing enables both tractable training and controlled generalization.
  arXiv  Detail & Related papers  (2024-07-24T12:00:30Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• A Prescriptive Dirichlet Power Allocation Policy with Deep Reinforcement Learning [6.003234406806134]
  In this work, we propose the Dirichlet policy for continuous allocation tasks and analyze the bias and variance of its policy gradients. We demonstrate that the Dirichlet policy is bias-free and provides significantly faster convergence and better performance than the Gaussian-softmax policy. The experimental results show the potential to prescribe optimal operation, improve the efficiency and sustainability of multi-power source systems.
  arXiv  Detail & Related papers  (2022-01-20T20:41:04Z)
• Distributed Adaptive Learning Under Communication Constraints [54.22472738551687]
  This work examines adaptive distributed learning strategies designed to operate under communication constraints. We consider a network of agents that must solve an online optimization problem from continual observation of streaming data.
  arXiv  Detail & Related papers  (2021-12-03T19:23:48Z)
• Verifiable Planning in Expected Reward Multichain MDPs [20.456052208569115]
  We explore the steady-state planning problem of deriving a decision-making policy for an agent. We prove that optimal solutions to the proposed programs yield stationary policies with rigorous guarantees of behavior.
  arXiv  Detail & Related papers  (2020-12-03T18:54:24Z)
• Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints [52.58352707495122]
  We present a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under uncertainty and multi-agent coordination. We validate our results over a wide range of simulations on two distinct domains: multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in a city.
  arXiv  Detail & Related papers  (2020-05-27T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.