DEPLOYR: A technical framework for deploying custom real-time machine learning models into the electronic medical record

• URL: http://arxiv.org/abs/2303.06269v1
• Date: Sat, 11 Mar 2023 01:11:24 GMT
• Title: DEPLOYR: A technical framework for deploying custom real-time machine learning models into the electronic medical record
• Abstract summary: We present DEPLOYR, a framework for enabling real-time deployment and monitoring of machine learning models. We discuss core functionality and design decisions, including mechanisms to trigger inference based on actions within EMR software. We demonstrate the use of DEPLOYR by silently deploying and prospectively evaluating twelve ML models triggered by clinician button-clicks in Stanford Health Care's production instance of Epic.
• Score: 5.660769142308798
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning (ML) applications in healthcare are extensively researched, but successful translations to the bedside are scant. Healthcare institutions are establishing frameworks to govern and promote the implementation of accurate, actionable and reliable models that integrate with clinical workflow. Such governance frameworks require an accompanying technical framework to deploy models in a resource efficient manner. Here we present DEPLOYR, a technical framework for enabling real-time deployment and monitoring of researcher created clinical ML models into a widely used electronic medical record (EMR) system. We discuss core functionality and design decisions, including mechanisms to trigger inference based on actions within EMR software, modules that collect real-time data to make inferences, mechanisms that close-the-loop by displaying inferences back to end-users within their workflow, monitoring modules that track performance of deployed models over time, silent deployment capabilities, and mechanisms to prospectively evaluate a deployed model's impact. We demonstrate the use of DEPLOYR by silently deploying and prospectively evaluating twelve ML models triggered by clinician button-clicks in Stanford Health Care's production instance of Epic. Our study highlights the need and feasibility for such silent deployment, because prospectively measured performance varies from retrospective estimates. By describing DEPLOYR, we aim to inform ML deployment best practices and help bridge the model implementation gap.

Related papers

• Future Querying: Can LLMs Serve as Implicit Medical World Models? [3.3341114691426257]
  We introduce future querying, a paradigm that probes whether large language models (LLMs) can function as implicit medical world models.<n>Our framework operates on unstructured clinical documentation using endpoint-agnostic training.<n>We show that small, locally fine-tuned open-weight models can match or approach larger proprietary systems.
  arXiv  Detail & Related papers  (2026-08-24T13:41:12Z)
• Volumetric Radiology AI in the Era of Multimodal Large Language Models [62.95443321443533]
  Volumetric radiology presents a fundamental representational mismatch.<n>Current large language models (MLLMs) are conditioned on selected two-dimensional (2D) images, compressed visual representations, or report-derived text.<n>Reliable radiology AI requires representations that preserve task-relevant 3D information and systems that can access, verify, and integrate this information.
  arXiv  Detail & Related papers  (2026-08-20T20:23:41Z)
• Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models [122.58252919699122]
  Mechanistic Interpretability (MI) has emerged as a vital approach to demystify the decision-making of Large Language Models (LLMs)<n>We present a practical survey structured around the pipeline: "Awesomeinterventionable-MI-Survey"
  arXiv  Detail & Related papers  (2026-01-20T14:23:23Z)
• An Agentic Model Context Protocol Framework for Medical Concept Standardization [5.12407270785129]
  We develop a zero-training, hallucination-preventive mapping system based on the Model Context Protocol (MCP)<n>The system enables explainable mapping and significantly improves efficiency and accuracy with minimal effort.
  arXiv  Detail & Related papers  (2025-09-04T02:32:22Z)
• Enhancing Cell Counting through MLOps: A Structured Approach for Automated Cell Analysis [5.615229139948217]
  This paper introduces Cell Counting Machine Learning Operations (CC-MLOps), a comprehensive framework that streamlines the integration of machine learning in cell counting. Through a practical use case, we demonstrate how MLOps principles can enhance reliability, reduce human error, and enable scalable Cell Counting solutions.
  arXiv  Detail & Related papers  (2025-04-28T13:53:50Z)
• Self-Healing Machine Learning: A Framework for Autonomous Adaptation in Real-World Environments [50.310636905746975]
  Real-world machine learning systems often encounter model performance degradation due to distributional shifts in the underlying data generating process. Existing approaches to addressing shifts, such as concept drift adaptation, are limited by their reason-agnostic nature. We propose self-healing machine learning (SHML) to overcome these limitations.
  arXiv  Detail & Related papers  (2024-10-31T20:05:51Z)
• R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models [50.19174067263255]
  We introduce prior preference learning techniques and self-revision schedules to help the agent excel in sparse-reward, continuous action, goal-based robotic control POMDP environments. We show that our agents offer improved performance over state-of-the-art models in terms of cumulative rewards, relative stability, and success rate.
  arXiv  Detail & Related papers  (2024-09-21T18:32:44Z)
• Clinical Validation of a Real-Time Machine Learning-based System for the Detection of Acute Myeloid Leukemia by Flow Cytometry [0.0]
  Machine-learning (ML) models in flow have the potential to reduce error rates, increase efficiency, and boost the efficiency of clinical labs. Few studies have described the clinical deployment of such models. We describe an ML model for detection of Acute Myeloid Leukemia (AML) along with the infrastructure supporting clinical implementation.
  arXiv  Detail & Related papers  (2024-09-17T16:53:47Z)
• MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications [2.838746648891565]
  We introduce MEDIC, a framework assessing Large Language Models (LLMs) across five critical dimensions of clinical competence. We apply MEDIC to evaluate LLMs on medical question-answering, safety, summarization, note generation, and other tasks. Results show performance disparities across model sizes, baseline vs medically finetuned models, and have implications on model selection for applications requiring specific model strengths.
  arXiv  Detail & Related papers  (2024-09-11T14:44:51Z)
• Benchmarks as Microscopes: A Call for Model Metrology [76.64402390208576]
  Modern language models (LMs) pose a new challenge in capability assessment. To be confident in our metrics, we need a new discipline of model metrology.
  arXiv  Detail & Related papers  (2024-07-22T17:52:12Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML [0.7982607013768545]
  Yet Another ICU Benchmark (YAIB) is a modular framework that allows researchers to define reproducible and comparable clinical ML experiments. YAIB supports most open-access ICU datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future ICU datasets. We demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance.
  arXiv  Detail & Related papers  (2023-06-08T11:16:20Z)
• Safe AI for health and beyond -- Monitoring to transform a health service [51.8524501805308]
  We will assess the infrastructure required to monitor the outputs of a machine learning algorithm. We will present two scenarios with examples of monitoring and updates of models.
  arXiv  Detail & Related papers  (2023-03-02T17:27:45Z)
• System Resilience through Health Monitoring and Reconfiguration [56.448036299746285]
  We demonstrate an end-to-end framework to improve the resilience of man-made systems to unforeseen events. The framework is based on a physics-based digital twin model and three modules tasked with real-time fault diagnosis, prognostics and reconfiguration.
  arXiv  Detail & Related papers  (2022-08-30T20:16:17Z)
• Enabling Un-/Semi-Supervised Machine Learning for MDSE of the Real-World CPS/IoT Applications [0.5156484100374059]
  We propose a novel approach to support domain-specific Model-Driven Software Engineering (MDSE) for the real-world use-case scenarios of smart Cyber-Physical Systems (CPS) and the Internet of Things (IoT) We argue that the majority of available data in the nature for Artificial Intelligence (AI) are unlabeled. Hence, unsupervised and/or semi-supervised ML approaches are the practical choices. Our proposed approach is fully implemented and integrated with an existing state-of-the-art MDSE tool to serve the CPS/IoT domain.
  arXiv  Detail & Related papers  (2021-07-06T15:51:39Z)
• Assessing YOLACT++ for real time and robust instance segmentation of medical instruments in endoscopic procedures [0.5735035463793008]
  Image-based tracking of laparoscopic instruments plays a fundamental role in computer and robotic-assisted surgeries. To date, most of the existing models for instance segmentation of medical instruments were based on two-stage detectors. We propose the addition of attention mechanisms to the YOLACT architecture that allows real-time instance segmentation of instruments.
  arXiv  Detail & Related papers  (2021-03-30T00:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.