The Silent Problem -- Machine Learning Model Failure -- How to Diagnose and Fix Ailing Machine Learning Models

• URL: http://arxiv.org/abs/2204.10227v1
• Date: Thu, 21 Apr 2022 16:05:42 GMT
• Title: The Silent Problem -- Machine Learning Model Failure -- How to Diagnose and Fix Ailing Machine Learning Models
• Abstract summary: The COVID-19 pandemic has dramatically changed how healthcare is delivered to patients. Analytical models that were trained and tested pre-pandemic may no longer be performing up to expectations. Model building must incorporate intentional resilience, the ability to offset and recover quickly from failure, and proactive robustness.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The COVID-19 pandemic has dramatically changed how healthcare is delivered to patients, how patients interact with healthcare providers, and how healthcare information is disseminated to both healthcare providers and patients. Analytical models that were trained and tested pre-pandemic may no longer be performing up to expectations, providing unreliable and irrelevant learning (ML) models given that ML depends on the basic principle that what happened in the past are likely to repeat in the future. ML faced to two important degradation principles, concept drift, when the underlying properties and characteristics of the variables change and data drift, when the data distributions, probabilities, co-variates, and other variable relationships change, both of which are prime culprits of model failure. Therefore, detecting and diagnosing drift in existing models is something that has become an imperative. And perhaps even more important is a shift in our mindset towards a conscious recognition that drift is inevitable, and model building must incorporate intentional resilience, the ability to offset and recover quickly from failure, and proactive robustness, avoiding failure by developing models that are less vulnerable to drift and disruption.

Related papers

• Bounding the Causal Impact of ML-assisted Decision-Making via Counterfactual Correctness [2.9267249513932008]
  We present a partial-identification approach to constructing bounds on the causal effect of a new model.<n>The core innovation in our approach is to leverage assumptions relating fine-grained predictive accuracy to downstream outcomes.
  arXiv  Detail & Related papers  (2026-07-23T20:45:10Z)
• Beyond Traditional Diagnostics: Transforming Patient-Side Information into Predictive Insights with Knowledge Graphs and Prototypes [55.310195121276074]
  We propose a Knowledge graph-enhanced, Prototype-aware, and Interpretable (KPI) framework to predict diseases.<n>It integrates structured and trusted medical knowledge into a unified disease knowledge graph, constructs clinically meaningful disease prototypes, and employs contrastive learning to enhance predictive accuracy.<n>It provides clinically valid explanations that closely align with patient narratives, highlighting its practical value for patient-centered healthcare delivery.
  arXiv  Detail & Related papers  (2025-12-09T05:37:54Z)
• From Physics to Machine Learning and Back: Part II - Learning and Observational Bias in PHM [52.64097278841485]
  Review examines how incorporating learning and observational biases through physics-informed modeling and data strategies can guide models toward physically consistent and reliable predictions.<n>Fast adaptation methods including meta-learning and few-shot learning are reviewed alongside domain generalization techniques.
  arXiv  Detail & Related papers  (2025-09-25T14:15:43Z)
• Is Uncertainty Quantification a Viable Alternative to Learned Deferral? [1.533133219129073]
  One aspect of AI safety is the models' ability to defer decisions to a human expert.<n>During clinical translation, models often face challenges such as data shift.<n>Uncertainty quantification methods may be a promising choice for AI deferral.
  arXiv  Detail & Related papers  (2025-08-04T11:37:59Z)
• Keeping Medical AI Healthy: A Review of Detection and Correction Methods for System Degradation [6.781778751487079]
  This review presents a forward-looking perspective on monitoring and maintaining the "health" of AI systems in healthcare.<n>We highlight the urgent need for continuous performance monitoring, early degradation detection, and effective self-correction mechanisms.<n>This work aims to guide the development of reliable, robust medical AI systems capable of sustaining safe, long-term deployment in dynamic clinical settings.
  arXiv  Detail & Related papers  (2025-06-20T19:22:07Z)
• Common Steps in Machine Learning Might Hinder The Explainability Aims in Medicine [0.0]
  This paper discusses the steps of the data preprocessing in machine learning and their impacts on the explainability and interpretability of the model. It is found the steps improve the accuracy of the model, but they might hinder the explainability of the model if they are not carefully considered especially in medicine.
  arXiv  Detail & Related papers  (2024-08-30T12:09:14Z)
• Advancing Brain Imaging Analysis Step-by-step via Progressive Self-paced Learning [0.5840945370755134]
  We introduce the Progressive Self-Paced Distillation (PSPD) framework, employing an adaptive and progressive pacing and distillation mechanism. We validate PSPD's efficacy and adaptability across various convolutional neural networks using the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset.
  arXiv  Detail & Related papers  (2024-07-23T02:26:04Z)
• MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation [58.93221876843639]
  This paper introduces a novel, end-to-end diffusion-based risk prediction model, named MedDiffusion. It enhances risk prediction performance by creating synthetic patient data during training to enlarge sample space. It discerns hidden relationships between patient visits using a step-wise attention mechanism, enabling the model to automatically retain the most vital information for generating high-quality data.
  arXiv  Detail & Related papers  (2023-10-04T01:36:30Z)
• Safe AI for health and beyond -- Monitoring to transform a health service [51.8524501805308]
  We will assess the infrastructure required to monitor the outputs of a machine learning algorithm. We will present two scenarios with examples of monitoring and updates of models.
  arXiv  Detail & Related papers  (2023-03-02T17:27:45Z)
• SPADE4: Sparsity and Delay Embedding based Forecasting of Epidemics [2.578242050187029]
  We propose Sparsity and Delay Embedding based Forecasting (SPADE4) for predicting epidemics. We show that our approach outperforms compartmental models when applied to both simulated and real data.
  arXiv  Detail & Related papers  (2022-11-11T23:39:48Z)
• Distillation to Enhance the Portability of Risk Models Across Institutions with Large Patient Claims Database [12.452703677540505]
  We investigate the practicality of model portability through a cross-site evaluation of readmission prediction models. We apply a recurrent neural network, augmented with self-attention and blended with expert features, to build readmission prediction models. Our experiments show that direct application of ML models trained at one institution and tested at another institution perform worse than models trained and tested at the same institution.
  arXiv  Detail & Related papers  (2022-07-06T05:26:32Z)
• Benchmarking Heterogeneous Treatment Effect Models through the Lens of Interpretability [82.29775890542967]
  Estimating personalized effects of treatments is a complex, yet pervasive problem. Recent developments in the machine learning literature on heterogeneous treatment effect estimation gave rise to many sophisticated, but opaque, tools. We use post-hoc feature importance methods to identify features that influence the model's predictions.
  arXiv  Detail & Related papers  (2022-06-16T17:59:05Z)
• What Do You See in this Patient? Behavioral Testing of Clinical NLP Models [69.09570726777817]
  We introduce an extendable testing framework that evaluates the behavior of clinical outcome models regarding changes of the input. We show that model behavior varies drastically even when fine-tuned on the same data and that allegedly best-performing models have not always learned the most medically plausible patterns.
  arXiv  Detail & Related papers  (2021-11-30T15:52:04Z)
• Learning transition times in event sequences: the Event-Based Hidden Markov Model of disease progression [4.12857285066818]
  We connect ideas from event-based and hidden Markov modelling to derive a new generative model of disease progression. Our model can infer the most likely group-level sequence and timing of events from limited datasets. We use clinical, imaging and biofluid data from the Alzheimer's Disease Neuroimaging Initiative to demonstrate the validity and utility of our model.
  arXiv  Detail & Related papers  (2020-11-02T15:13:03Z)
• An Optimal Control Approach to Learning in SIDARTHE Epidemic model [67.22168759751541]
  We propose a general approach for learning time-variant parameters of dynamic compartmental models from epidemic data. We forecast the epidemic evolution in Italy and France.
  arXiv  Detail & Related papers  (2020-10-28T10:58:59Z)
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data [81.00385374948125]
  We present UNcertaInTy-based hEalth risk prediction (UNITE) model. UNITE provides accurate disease risk prediction and uncertainty estimation leveraging multi-sourced health data. We evaluate UNITE on real-world disease risk prediction tasks: nonalcoholic fatty liver disease (NASH) and Alzheimer's disease (AD) UNITE achieves up to 0.841 in F1 score for AD detection, up to 0.609 in PR-AUC for NASH detection, and outperforms various state-of-the-art baselines by up to $19%$ over the best baseline.
  arXiv  Detail & Related papers  (2020-10-22T02:28:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.