CheXstray: Real-time Multi-Modal Data Concordance for Drift Detection in Medical Imaging AI

• URL: http://arxiv.org/abs/2202.02833v1
• Date: Sun, 6 Feb 2022 18:58:35 GMT
• Title: CheXstray: Real-time Multi-Modal Data Concordance for Drift Detection in Medical Imaging AI
• Authors: Arjun Soin, Jameson Merkow, Jin Long, Joesph Paul Cohen, Smitha Saligrama, Stephen Kaiser, Steven Borg, Ivan Tarapov and Matthew P Lungren
• Abstract summary: We build and test a medical imaging AI drift monitoring workflow that tracks data and model drift without contemporaneous ground truth. Key contributions include (1) proof-of-concept for medical imaging drift detection including use of VAE and domain specific statistical methods. This work has important implications for addressing the translation gap related to continuous medical imaging AI model monitoring in dynamic healthcare environments.
• Score: 1.359138408203412
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rapidly expanding Clinical AI applications worldwide have the potential to impact to all areas of medical practice. Medical imaging applications constitute a vast majority of approved clinical AI applications. Though healthcare systems are eager to adopt AI solutions a fundamental question remains: \textit{what happens after the AI model goes into production?} We use the CheXpert and PadChest public datasets to build and test a medical imaging AI drift monitoring workflow that tracks data and model drift without contemporaneous ground truth. We simulate drift in multiple experiments to compare model performance with our novel multi-modal drift metric, which uses DICOM metadata, image appearance representation from a variational autoencoder (VAE), and model output probabilities as input. Through experimentation, we demonstrate a strong proxy for ground truth performance using unsupervised distributional shifts in relevant metadata, predicted probabilities, and VAE latent representation. Our key contributions include (1) proof-of-concept for medical imaging drift detection including use of VAE and domain specific statistical methods (2) a multi-modal methodology for measuring and unifying drift metrics (3) new insights into the challenges and solutions for observing deployed medical imaging AI (4) creation of open-source tools enabling others to easily run their own workflows or scenarios. This work has important implications for addressing the translation gap related to continuous medical imaging AI model monitoring in dynamic healthcare environments.

Related papers

• Scalable Drift Monitoring in Medical Imaging AI [37.1899538374058]
  We develop MMC+, an enhanced framework for scalable drift monitoring. It builds upon the CheXstray framework that introduced real-time drift detection for medical imaging AI models. MMC+ offers a reliable and cost-effective alternative to continuous performance monitoring.
  arXiv  Detail & Related papers  (2024-10-17T02:57:35Z)
• MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild [81.32127423981426]
  Multimodal emotion recognition based on audio and video data is important for real-world applications. Recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. We propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders.
  arXiv  Detail & Related papers  (2024-04-13T13:39:26Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• VISION-MAE: A Foundation Model for Medical Image Segmentation and Classification [36.8105960525233]
  We present a novel foundation model, VISION-MAE, specifically designed for medical imaging. VISION-MAE is trained on a dataset of 2.5 million unlabeled images from various modalities. It is then adapted to classification and segmentation tasks using explicit labels.
  arXiv  Detail & Related papers  (2024-02-01T21:45:12Z)
• BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors [8.695342954247606]
  We present BenchMD, a benchmark that tests how well unified, modality-agnostic methods, including architectures and training techniques, perform on a diverse array of medical tasks. Our baseline results demonstrate that no unified learning technique achieves strong performance across all modalities, leaving ample room for improvement on the benchmark.
  arXiv  Detail & Related papers  (2023-04-17T17:59:26Z)
• Robust and Efficient Medical Imaging with Self-Supervision [80.62711706785834]
  We present REMEDIS, a unified representation learning strategy to improve robustness and data-efficiency of medical imaging AI. We study a diverse range of medical imaging tasks and simulate three realistic application scenarios using retrospective data.
  arXiv  Detail & Related papers  (2022-05-19T17:34:18Z)
• DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations [119.1953397679783]
  We focus on advancing the state-of-the-art in interpreting multimodal models. Our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models.
  arXiv  Detail & Related papers  (2022-03-03T20:52:47Z)
• MMLN: Leveraging Domain Knowledge for Multimodal Diagnosis [10.133715767542386]
  We propose a knowledge-driven and data-driven framework for lung disease diagnosis. We formulate diagnosis rules according to authoritative clinical medicine guidelines and learn the weights of rules from text data. A multimodal fusion consisting of text and image data is designed to infer the marginal probability of lung disease.
  arXiv  Detail & Related papers  (2022-02-09T04:12:30Z)
• Cross-Modal Information Maximization for Medical Imaging: CMIM [62.28852442561818]
  In hospitals, data are siloed to specific information systems that make the same information available under different modalities. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time. We propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time.
  arXiv  Detail & Related papers  (2020-10-20T20:05:35Z)
• Universal Model for Multi-Domain Medical Image Retrieval [88.67940265012638]
  Medical Image Retrieval (MIR) helps doctors quickly find similar patients' data. MIR is becoming increasingly helpful due to the wide use of digital imaging modalities. However, the popularity of various digital imaging modalities in hospitals also poses several challenges to MIR.
  arXiv  Detail & Related papers  (2020-07-14T23:22:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.