Monitoring Deployed AI Systems in Health Care

• URL: http://arxiv.org/abs/2512.09048v1
• Date: Tue, 09 Dec 2025 19:06:48 GMT
• Title: Monitoring Deployed AI Systems in Health Care
• Abstract summary: Post-deployment monitoring of artificial intelligence (AI) systems in health care is essential to ensure their safety, quality, and sustained benefit.<n>This framework is organized around three complementary principles: system integrity, performance, and impact.<n>It offers a practical template and starting point for health systems seeking to ensure that AI deployments remain safe and effective over time.
• Score: 8.240297189233528
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Post-deployment monitoring of artificial intelligence (AI) systems in health care is essential to ensure their safety, quality, and sustained benefit-and to support governance decisions about which systems to update, modify, or decommission. Motivated by these needs, we developed a framework for monitoring deployed AI systems grounded in the mandate to take specific actions when they fail to behave as intended. This framework, which is now actively used at Stanford Health Care, is organized around three complementary principles: system integrity, performance, and impact. System integrity monitoring focuses on maximizing system uptime, detecting runtime errors, and identifying when changes to the surrounding IT ecosystem have unintended effects. Performance monitoring focuses on maintaining accurate system behavior in the face of changing health care practices (and thus input data) over time. Impact monitoring assesses whether a deployed system continues to have value in the form of benefit to clinicians and patients. Drawing on examples of deployed AI systems at our academic medical center, we provide practical guidance for creating monitoring plans based on these principles that specify which metrics to measure, when those metrics should be reviewed, who is responsible for acting when metrics change, and what concrete follow-up actions should be taken-for both traditional and generative AI. We also discuss challenges to implementing this framework, including the effort and cost of monitoring for health systems with limited resources and the difficulty of incorporating data-driven monitoring practices into complex organizations where conflicting priorities and definitions of success often coexist. This framework offers a practical template and starting point for health systems seeking to ensure that AI deployments remain safe and effective over time.

Related papers

• AI Deployment Accountability Engineering: A Vision for Accountable AI in Safety-Critical Socio-Technical Systems [14.280538199149634]
  This vision paper introduces AI Deployment Accountability Engineering (ADAE)<n>ADAE is a proposed AI engineering subdiscipline concerned with establishing measurable, continuous, and actionable accountability for deployed AI systems.<n>We articulate a research agenda built around four interconnected pillars: structured discovery of context-dependent failure modes, privacy-preserving accountability measurement, system-level risk analysis for agentic AI, and translation of technical failures into operational, and institutional risks.
  arXiv  Detail & Related papers  (2026-09-13T15:18:41Z)
• ETHOS: Towards a Modular Ethics Framework for Clinical Multi-Agent Systems [7.256126882767553]
  We present ETHOS, a governance meta-agent that can be integrated with any existing multi-agent system.<n>We demonstrate ETHOS within a hepatology clinical decision-support MAS.<n>Results show that ETHOS improves decision reliability by detecting incomplete, inconsistent, or out-of-scope evidence.
  arXiv  Detail & Related papers  (2026-08-15T21:56:23Z)
• Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels [53.63220028820581]
  Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction.<n>We define trustworthy embodied intelligence as the sustained capacity to execute specified tasks reliably.
  arXiv  Detail & Related papers  (2026-07-28T17:50:44Z)
• Towards Auditing AI Systems in the Wild [3.9549290063964313]
  We argue for the development of principled auditing frameworks that monitor deployed AI systems throughout their lifecycle.<n>This perspective highlights the need for uncertainty-aware monitoring methods, socio-technical specifications of audit criteria, and auditing infrastructures.
  arXiv  Detail & Related papers  (2026-06-15T23:52:11Z)
• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• When AI Gets it Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems [0.0]
  In high-risk domains such as medication management, even a single incorrect recommendation can result in severe patient harm.<n>This paper examines the reliability of AI-assisted medication systems by focusing on system failures and their potential clinical consequences.
  arXiv  Detail & Related papers  (2026-04-01T22:53:17Z)
• Digital health transformation in Quebec: assessment of interoperability and governance strategies [45.88028371034407]
  interoperability remains a major challenge within Quebec's Health and Social Services Network.<n>This paper assessed how Quebec's Plan sante addressed interoperability challenges using the dimensions from the Healthcare Information and Management Systems Society (HIMSS)<n>Overall, while the Plan sante demonstrates a clear commitment to technological modernization, it does not fully address the interoperability multidimensional nature.
  arXiv  Detail & Related papers  (2025-12-18T23:33:05Z)
• The SMART+ Framework for AI Systems [0.2039123720459736]
  Artificial Intelligence (AI) systems are now an integral part of multiple industries.<n>In clinical research, AI supports automated adverse event detection in clinical trials, patient eligibility screening for protocol enrollment, and data quality validation.<n>Beyond healthcare, AI is transforming finance through real-time fraud detection, automated loan risk assessment, and algorithmic decision-making.
  arXiv  Detail & Related papers  (2025-12-09T13:33:14Z)
• Perspectives on a Reliability Monitoring Framework for Agentic AI Systems [5.539407031861404]
  We derive the main reliability challenges of agentic AI systems during operation based on their characteristics.<n>We propose a two-layered reliability monitoring framework for agentic AI systems.
  arXiv  Detail & Related papers  (2025-11-12T10:19:17Z)
• Keeping Medical AI Healthy: A Review of Detection and Correction Methods for System Degradation [6.781778751487079]
  This review presents a forward-looking perspective on monitoring and maintaining the "health" of AI systems in healthcare.<n>We highlight the urgent need for continuous performance monitoring, early degradation detection, and effective self-correction mechanisms.<n>This work aims to guide the development of reliable, robust medical AI systems capable of sustaining safe, long-term deployment in dynamic clinical settings.
  arXiv  Detail & Related papers  (2025-06-20T19:22:07Z)
• Beyond One-Time Validation: A Framework for Adaptive Validation of Prognostic and Diagnostic AI-based Medical Devices [55.319842359034546]
  Existing approaches often fall short in addressing the complexity of practically deploying these devices. The presented framework emphasizes the importance of repeating validation and fine-tuning during deployment. It is positioned within the current US and EU regulatory landscapes.
  arXiv  Detail & Related papers  (2024-09-07T11:13:52Z)
• Trustworthy and Practical AI for Healthcare: A Guided Deferral System with Large Language Models [1.2281181385434294]
  Large language models (LLMs) offer a valuable technology for various applications in healthcare.<n>Their tendency to hallucinate and the existing reliance on proprietary systems pose challenges in environments concerning critical decision-making.<n>This paper presents a novel HAIC guided deferral system that can simultaneously parse medical reports for disorder classification, and defer uncertain predictions with intelligent guidance to humans.
  arXiv  Detail & Related papers  (2024-06-11T12:41:54Z)
• Deep Reinforcement Learning Empowered Activity-Aware Dynamic Health Monitoring Systems [69.41229290253605]
  Existing monitoring approaches were designed on the premise that medical devices track several health metrics concurrently. This means that they report all relevant health values within that scope, which can result in excess resource use and the gathering of extraneous data. We propose Dynamic Activity-Aware Health Monitoring strategy (DActAHM) for striking a balance between optimal monitoring performance and cost efficiency.
  arXiv  Detail & Related papers  (2024-01-19T16:26:35Z)
• Evaluating General-Purpose AI with Psychometrics [43.85432514910491]
  We discuss the need for a comprehensive and accurate evaluation of general-purpose AI systems such as large language models. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation.
  arXiv  Detail & Related papers  (2023-10-25T05:38:38Z)
• Organizational Governance of Emerging Technologies: AI Adoption in Healthcare [43.02293389682218]
  The Health AI Partnership aims to better define the requirements for adequate organizational governance of AI systems in healthcare settings. This is one of the most detailed qualitative analyses to date of the current governance structures and processes involved in AI adoption by health systems in the United States. We hope these findings can inform future efforts to build capabilities to promote the safe, effective, and responsible adoption of emerging technologies in healthcare.
  arXiv  Detail & Related papers  (2023-04-25T18:30:47Z)
• I-Health: Leveraging Edge Computing and Blockchain for Epidemic Management [36.55809341110476]
  Epidemic situations demand intensive data collection and management from different locations/entities within a strict time constraint. This paper proposes an Intelligent-Health (I-Health) system that aims to aggregate diverse e-health entities in a unique national healthcare system. In particular, we design an automated patients monitoring scheme, at the edge, which enables the prompt discovery, remote monitoring, and fast emergency response.
  arXiv  Detail & Related papers  (2020-12-18T23:41:00Z)
• Monitoring and Diagnosability of Perception Systems [21.25149064251918]
  Perception is a critical component of high-integrity applications of robotics and autonomous systems, such as self-driving cars. Despite the paramount importance of perception systems, there is no formal approach for system-level monitoring. We propose a mathematical model for runtime monitoring and fault detection of perception systems.
  arXiv  Detail & Related papers  (2020-05-24T18:09:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.