Explainability of Traditional and Deep Learning Models on Longitudinal Healthcare Records

• URL: http://arxiv.org/abs/2211.12002v1
• Date: Tue, 22 Nov 2022 04:39:17 GMT
• Title: Explainability of Traditional and Deep Learning Models on Longitudinal Healthcare Records
• Authors: Lin Lee Cheong (1), Tesfagabir Meharizghi (1), Wynona Black (2), Yang Guang (1) and Weilin Meng (2) ((1) Amazon ML Solutions Lab, (2) Merck & Co., Inc)
• Abstract summary: Rigorous evaluation of explainability is often missing, as comparisons between models and various explainability methods have not been well-studied. Our work is one of the first to evaluate explainability performance between and within traditional (XGBoost) and deep learning (LSTM with Attention) models on both a global and individual per-prediction level.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in deep learning have led to interest in training deep learning models on longitudinal healthcare records to predict a range of medical events, with models demonstrating high predictive performance. Predictive performance is necessary but insufficient, however, with explanations and reasoning from models required to convince clinicians for sustained use. Rigorous evaluation of explainability is often missing, as comparisons between models (traditional versus deep) and various explainability methods have not been well-studied. Furthermore, ground truths needed to evaluate explainability can be highly subjective depending on the clinician's perspective. Our work is one of the first to evaluate explainability performance between and within traditional (XGBoost) and deep learning (LSTM with Attention) models on both a global and individual per-prediction level on longitudinal healthcare data. We compared explainability using three popular methods: 1) SHapley Additive exPlanations (SHAP), 2) Layer-Wise Relevance Propagation (LRP), and 3) Attention. These implementations were applied on synthetically generated datasets with designed ground-truths and a real-world medicare claims dataset. We showed that overall, LSTMs with SHAP or LRP provides superior explainability compared to XGBoost on both the global and local level, while LSTM with dot-product attention failed to produce reasonable ones. With the explosion of the volume of healthcare data and deep learning progress, the need to evaluate explainability will be pivotal towards successful adoption of deep learning models in healthcare settings.

Related papers

• Towards Interpretable End-Stage Renal Disease (ESRD) Prediction: Utilizing Administrative Claims Data with Explainable AI Techniques [6.417777780911223]
  This study explores the potential of utilizing administrative claims data, combined with advanced machine learning and deep learning techniques, to predict the progression of Chronic Kidney Disease (CKD) to End-Stage Renal Disease (ESRD) We analyze a comprehensive, 10-year dataset provided by a major health insurance organization to develop prediction models for multiple observation windows using traditional machine learning methods such as Random Forest and XGBoost as well as deep learning approaches such as Long Short-Term Memory (LSTM) networks. Our findings demonstrate that the LSTM model, particularly with a 24-month observation window, exhibits superior performance in predicting ESRD progression,
  arXiv  Detail & Related papers  (2024-09-18T16:03:57Z)
• Comparative Analysis of LSTM Neural Networks and Traditional Machine Learning Models for Predicting Diabetes Patient Readmission [0.0]
  This study uses the Diabetes 130-US Hospitals dataset for analysis and prediction of readmission patients by various machine learning models. LightGBM turned out to be the best traditional model, while XGBoost was the runner-up. This study demonstrates that model selection, validation, and interpretability are key steps in predictive healthcare modeling.
  arXiv  Detail & Related papers  (2024-06-28T15:06:22Z)
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models [57.43276586087863]
  Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs. Existing benchmarks are often limited in scope, focusing mainly on object hallucinations. We introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases.
  arXiv  Detail & Related papers  (2024-04-22T04:49:22Z)
• Continual Learning with Pre-Trained Models: A Survey [61.97613090666247]
  Continual Learning aims to overcome the catastrophic forgetting of former knowledge when learning new ones. This paper presents a comprehensive survey of the latest advancements in PTM-based CL.
  arXiv  Detail & Related papers  (2024-01-29T18:27:52Z)
• Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models [49.95603725998561]
  We propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model.
  arXiv  Detail & Related papers  (2023-10-04T21:57:09Z)
• Can Attention Be Used to Explain EHR-Based Mortality Prediction Tasks: A Case Study on Hemorrhagic Stroke [33.08002675910282]
  Stroke is a significant cause of mortality and morbidity, necessitating early predictive strategies to minimize risks. Traditional methods for evaluating patients have limited accuracy and interpretability. This paper proposes an interpretable, attention-based transformer model for early stroke mortality prediction.
  arXiv  Detail & Related papers  (2023-08-04T04:28:07Z)
• TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network [54.332862955411656]
  Clinical trials are critical for drug development but often suffer from expensive and inefficient patient recruitment. In recent years, machine learning models have been proposed for speeding up patient recruitment via automatically matching patients with clinical trials. We introduce a dynamic tree-based memory network model named TREEMENT to provide accurate and interpretable patient trial matching.
  arXiv  Detail & Related papers  (2023-07-19T12:35:09Z)
• How robust are pre-trained models to distribution shift? [82.08946007821184]
  We show how spurious correlations affect the performance of popular self-supervised learning (SSL) and auto-encoder based models (AE) We develop a novel evaluation scheme with the linear head trained on out-of-distribution (OOD) data, to isolate the performance of the pre-trained models from a potential bias of the linear head used for evaluation.
  arXiv  Detail & Related papers  (2022-06-17T16:18:28Z)
• IA-GCN: Interpretable Attention based Graph Convolutional Network for Disease prediction [47.999621481852266]
  We propose an interpretable graph learning-based model which interprets the clinical relevance of the input features towards the task. In a clinical scenario, such a model can assist the clinical experts in better decision-making for diagnosis and treatment planning. Our proposed model shows superior performance with respect to compared methods with an increase in an average accuracy of 3.2% for Tadpole, 1.6% for UKBB Gender, and 2% for the UKBB Age prediction task.
  arXiv  Detail & Related papers  (2021-03-29T13:04:02Z)
• Explaining Deep Learning Models for Structured Data using Layer-Wise Relevance Propagation [0.0]
  Layer-wise Relevance (LRP), an established explainability technique developed for deep models in computer vision, provides intuitive human-readable heat maps of input images. We show how LRPis more effective than traditional explainability concepts of Local Interpretable Model-agnostic Ex-planations (LIME) and Shapley Additive Explanations (SHAP) for explainability.
  arXiv  Detail & Related papers  (2020-11-26T18:34:21Z)
• Enhancing the Interpretability of Deep Models in Heathcare Through Attention: Application to Glucose Forecasting for Diabetic People [4.692400531340393]
  We evaluate the RETAIN model on the type-2 IDIAB and the type-1 OhioT1DM datasets. We show that the RETAIN model offers a very good compromise between accuracy and interpretability.
  arXiv  Detail & Related papers  (2020-09-08T13:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.