CheXternal: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays and External Clinical Settings

• URL: http://arxiv.org/abs/2102.08660v1
• Date: Wed, 17 Feb 2021 09:58:14 GMT
• Title: CheXternal: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays and External Clinical Settings
• Authors: Pranav Rajpurkar, Anirudh Joshi, Anuj Pareek, Andrew Y. Ng, Matthew P. Lungren
• Abstract summary: We measured the diagnostic performance for 8 different chest X-ray models when applied to smartphone photos of chest X-rays and external datasets. We found that on photos of chest X-rays, all 8 models experienced a statistically significant drop in task performance. Some chest X-ray models, under clinically relevant distribution shifts, were comparable to radiologists while other models were not.
• Score: 6.133159722996137
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in training deep learning models have demonstrated the potential to provide accurate chest X-ray interpretation and increase access to radiology expertise. However, poor generalization due to data distribution shifts in clinical settings is a key barrier to implementation. In this study, we measured the diagnostic performance for 8 different chest X-ray models when applied to (1) smartphone photos of chest X-rays and (2) external datasets without any finetuning. All models were developed by different groups and submitted to the CheXpert challenge, and re-applied to test datasets without further tuning. We found that (1) on photos of chest X-rays, all 8 models experienced a statistically significant drop in task performance, but only 3 performed significantly worse than radiologists on average, and (2) on the external set, none of the models performed statistically significantly worse than radiologists, and five models performed statistically significantly better than radiologists. Our results demonstrate that some chest X-ray models, under clinically relevant distribution shifts, were comparable to radiologists while other models were not. Future work should investigate aspects of model training procedures and dataset collection that influence generalization in the presence of data distribution shifts.

Related papers

• SynthEnsemble: A Fusion of CNN, Vision Transformer, and Hybrid Models for Multi-Label Chest X-Ray Classification [0.6218519716921521]
  We employ deep learning techniques to identify patterns in chest X-rays that correspond to different diseases. The best individual model was the CoAtNet, which achieved an area under the receiver operating characteristic curve (AUROC) of 84.2%.
  arXiv  Detail & Related papers  (2023-11-13T21:07:07Z)
• Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study [75.05049024176584]
  We present a benchmark study of the long-tailed learning problem in the specific domain of thorax diseases on chest X-rays. We focus on learning from naturally distributed chest X-ray data, optimizing classification accuracy over not only the common "head" classes, but also the rare yet critical "tail" classes. The benchmark consists of two chest X-ray datasets for 19- and 20-way thorax disease classification, containing classes with as many as 53,000 and as few as 7 labeled training images.
  arXiv  Detail & Related papers  (2022-08-29T04:34:15Z)
• The pitfalls of using open data to develop deep learning solutions for COVID-19 detection in chest X-rays [64.02097860085202]
  Deep learning models have been developed to identify COVID-19 from chest X-rays. Results have been exceptional when training and testing on open-source data. Data analysis and model evaluations show that the popular open-source dataset COVIDx is not representative of the real clinical problem.
  arXiv  Detail & Related papers  (2021-09-14T10:59:11Z)
• Contrastive Attention for Automatic Chest X-ray Report Generation [124.60087367316531]
  In most cases, the normal regions dominate the entire chest X-ray image, and the corresponding descriptions of these normal regions dominate the final report. We propose Contrastive Attention (CA) model, which compares the current input image with normal images to distill the contrastive information. We achieve the state-of-the-art results on the two public datasets.
  arXiv  Detail & Related papers  (2021-06-13T11:20:31Z)
• CheXbreak: Misclassification Identification for Deep Learning Models Interpreting Chest X-rays [5.263502842508203]
  We first investigate whether there are patient subgroups that chest x-ray models are likely to misclassify. Patient age and the radiographic finding of lung lesion or pneumothorax are statistically relevant features for predicting misclassification for some chest x-ray models. We develop misclassification predictors on chest x-ray models using their outputs and clinical features.
  arXiv  Detail & Related papers  (2021-03-18T00:30:19Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• CheXphotogenic: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays [4.396061096553544]
  We measured the diagnostic performance for 8 different chest x-ray models when applied to photos of chest x-rays. Several models had a drop in performance when applied to photos of chest x-rays, but even with this drop, some models still performed comparably to radiologists.
  arXiv  Detail & Related papers  (2020-11-12T00:16:51Z)
• Learning Invariant Feature Representation to Improve Generalization across Chest X-ray Datasets [55.06983249986729]
  We show that a deep learning model performing well when tested on the same dataset as training data starts to perform poorly when it is tested on a dataset from a different source. By employing an adversarial training strategy, we show that a network can be forced to learn a source-invariant representation.
  arXiv  Detail & Related papers  (2020-08-04T07:41:15Z)
• Deep Mining External Imperfect Data for Chest X-ray Disease Screening [57.40329813850719]
  We argue that incorporating an external CXR dataset leads to imperfect training data, which raises the challenges. We formulate the multi-label disease classification problem as weighted independent binary tasks according to the categories. Our framework simultaneously models and tackles the domain and label discrepancies, enabling superior knowledge mining ability.
  arXiv  Detail & Related papers  (2020-06-06T06:48:40Z)
• Context Learning for Bone Shadow Exclusion in CheXNet Accuracy Improvement [14.18754838863113]
  After ChestX-ray14 dataset containing over 100,000 frontal-view X-ray images of 14 diseases was released, several models were proposed with high accuracy. We develop a work flow for lung disease diagnosis in chest X-ray images, which can improve the average AUROC of the state-of-the-art model from 0.8414 to 0.8445.
  arXiv  Detail & Related papers  (2020-05-13T07:29:03Z)
• CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting [4.781964929315763]
  We examine the performance of the top 10 performing models on the CheXpert challenge leaderboard. We find that the top 10 chest x-ray models achieve an average AUC of 0.851 on the task of detecting TB on two public TB datasets. Second, we find that the average performance of the models on photos of x-rays (AUC = 0.916) is similar to their performance on the original chest x-ray images. Third, we find that the models tested on an external dataset either perform comparably to or exceed the average performance of radiologists.
  arXiv  Detail & Related papers  (2020-02-26T09:44:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.