Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation

• URL: http://arxiv.org/abs/2409.10526v1
• Date: Fri, 30 Aug 2024 15:13:58 GMT
• Title: Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation
• Authors: Anna L. Trella, Susobhan Ghosh, Erin E. Bonar, Lara Coughlin, Finale Doshi-Velez, Yongyi Guo, Pei-Yao Hung, Inbal Nahum-Shani, Vivek Shetty, Maureen Walton, Iris Yan, Kelly W. Zhang, Susan A. Murphy,
• Abstract summary: This paper provides guidelines for enabling effective monitoring of online decision-making algorithms. We discuss our experience in monitoring online decision-making algorithms in two digital intervention clinical trials.
• Score: 18.712461287884917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online AI decision-making algorithms are increasingly used by digital interventions to dynamically personalize treatment to individuals. These algorithms determine, in real-time, the delivery of treatment based on accruing data. The objective of this paper is to provide guidelines for enabling effective monitoring of online decision-making algorithms with the goal of (1) safeguarding individuals and (2) ensuring data quality. We elucidate guidelines and discuss our experience in monitoring online decision-making algorithms in two digital intervention clinical trials (Oralytics and MiWaves). Our guidelines include (1) developing fallback methods, pre-specified procedures executed when an issue occurs, and (2) identifying potential issues categorizing them by severity (red, yellow, and green). Across both trials, the monitoring systems detected real-time issues such as out-of-memory issues, database timeout, and failed communication with an external source. Fallback methods prevented participants from not receiving any treatment during the trial and also prevented the use of incorrect data in statistical analyses. These trials provide case studies for how health scientists can build monitoring systems for their digital intervention. Without these algorithm monitoring systems, critical issues would have gone undetected and unresolved. Instead, these monitoring systems safeguarded participants and ensured the quality of the resulting data for updating the intervention and facilitating scientific discovery. These monitoring guidelines and findings give digital intervention teams the confidence to include online decision-making algorithms in digital interventions.

Related papers

• Monitoring Fidelity of Online Reinforcement Learning Algorithms in Clinical Trials [20.944037982124037]
  This paper proposes algorithm fidelity as a critical requirement for deploying online RL algorithms in clinical trials. We present a framework for pre-deployment planning and real-time monitoring to help algorithm developers and clinical researchers ensure algorithm fidelity.
  arXiv  Detail & Related papers  (2024-02-26T20:19:14Z)
• Online Decision Mediation [72.80902932543474]
  Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior. In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
  arXiv  Detail & Related papers  (2023-10-28T05:59:43Z)
• Influence of the algorithm's reliability and transparency in the user's decision-making process [0.0]
  We conduct an online empirical study with 61 participants to find out how the change in transparency and reliability of an algorithm could impact users' decision-making process. The results indicate that people show at least moderate confidence in the decisions of the algorithm even when the reliability is bad.
  arXiv  Detail & Related papers  (2023-07-13T03:13:49Z)
• Contextual Bandits with Budgeted Information Reveal [3.861395476387163]
  Contextual bandit algorithms are commonly used in digital health to recommend personalized treatments. To ensure the effectiveness of the treatments, patients are often requested to take actions that have no immediate benefit to them. We introduce a novel optimization and learning algorithm to address this problem.
  arXiv  Detail & Related papers  (2023-05-29T16:18:28Z)
• Safe AI for health and beyond -- Monitoring to transform a health service [51.8524501805308]
  We will assess the infrastructure required to monitor the outputs of a machine learning algorithm. We will present two scenarios with examples of monitoring and updates of models.
  arXiv  Detail & Related papers  (2023-03-02T17:27:45Z)
• A Survey on Computer Vision based Human Analysis in the COVID-19 Era [58.79053747159797]
  The emergence of COVID-19 has had a global and profound impact, not only on society as a whole, but also on the lives of individuals. Various prevention measures were introduced around the world to limit the transmission of the disease, including face masks, mandates for social distancing and regular disinfection in public spaces, and the use of screening applications. These developments triggered the need for novel and improved computer vision techniques capable of (i) providing support to the prevention measures through an automated analysis of visual data, on the one hand, and (ii) facilitating normal operation of existing vision-based services, such as biometric authentication
  arXiv  Detail & Related papers  (2022-11-07T17:20:39Z)
• Federated Offline Reinforcement Learning [55.326673977320574]
  We propose a multi-site Markov decision process model that allows for both homogeneous and heterogeneous effects across sites. We design the first federated policy optimization algorithm for offline RL with sample complexity. We give a theoretical guarantee for the proposed algorithm, where the suboptimality for the learned policies is comparable to the rate as if data is not distributed.
  arXiv  Detail & Related papers  (2022-06-11T18:03:26Z)
• PosePipe: Open-Source Human Pose Estimation Pipeline for Clinical Research [0.0]
  We develop a human pose estimation pipeline that facilitates running state-of-the-art algorithms on data acquired in clinical context. Our goal in this work is not to train new algorithms, but to advance the use of cutting-edge human pose estimation algorithms for clinical and translation research.
  arXiv  Detail & Related papers  (2022-03-16T17:54:37Z)
• System Cards for AI-Based Decision-Making for Public Policy [5.076419064097733]
  This work proposes a system accountability benchmark for formal audits of artificial intelligence-based decision-aiding systems. It consists of 56 criteria organized within a four-by-four matrix composed of rows focused on (i) data, (ii) model, (iii) code, (iv) system, and columns focused on (a) development, (b) assessment, (c) mitigation, and (d) assurance.
  arXiv  Detail & Related papers  (2022-03-01T18:56:45Z)
• Adherence Forecasting for Guided Internet-Delivered Cognitive Behavioral Therapy: A Minimally Data-Sensitive Approach [59.535699822923]
  Internet-delivered psychological treatments (IDPT) are seen as an effective and scalable pathway to improving the accessibility of mental healthcare. This work proposes a deep-learning approach to perform automatic adherence forecasting, while relying on minimally sensitive login/logout data. The proposed Self-Attention Network achieved over 70% average balanced accuracy, when only 1/3 of the treatment duration had elapsed.
  arXiv  Detail & Related papers  (2022-01-11T13:55:57Z)
• DTR Bandit: Learning to Make Response-Adaptive Decisions With Low Regret [59.81290762273153]
  Dynamic treatment regimes (DTRs) are personalized, adaptive, multi-stage treatment plans that adapt treatment decisions to an individual's initial features and to intermediate outcomes and features at each subsequent stage. We propose a novel algorithm that, by carefully balancing exploration and exploitation, is guaranteed to achieve rate-optimal regret when the transition and reward models are linear.
  arXiv  Detail & Related papers  (2020-05-06T13:03:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.