Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset

• URL: http://arxiv.org/abs/2505.23030v1
• Date: Thu, 29 May 2025 03:16:18 GMT
• Title: Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset
• Abstract summary: This study compares four commercial clinical NLP systems for entity extraction and assertion detection.<n>CheXpert and CheXbert, two dedicated chest radiograph report labelers, were evaluated on the same task.<n>We analyzed 95,008 pediatric CXR reports from a large academic pediatric hospital.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: General-purpose clinical natural language processing (NLP) tools are increasingly used for the automatic labeling of clinical reports. However, independent evaluations for specific tasks, such as pediatric chest radiograph (CXR) report labeling, are limited. This study compares four commercial clinical NLP systems - Amazon Comprehend Medical (AWS), Google Healthcare NLP (GC), Azure Clinical NLP (AZ), and SparkNLP (SP) - for entity extraction and assertion detection in pediatric CXR reports. Additionally, CheXpert and CheXbert, two dedicated chest radiograph report labelers, were evaluated on the same task using CheXpert-defined labels. We analyzed 95,008 pediatric CXR reports from a large academic pediatric hospital. Entities and assertion statuses (positive, negative, uncertain) from the findings and impression sections were extracted by the NLP systems, with impression section entities mapped to 12 disease categories and a No Findings category. CheXpert and CheXbert extracted the same 13 categories. Outputs were compared using Fleiss Kappa and accuracy against a consensus pseudo-ground truth. Significant differences were found in the number of extracted entities and assertion distributions across NLP systems. SP extracted 49,688 unique entities, GC 16,477, AZ 31,543, and AWS 27,216. Assertion accuracy across models averaged around 62%, with SP highest (76%) and AWS lowest (50%). CheXpert and CheXbert achieved 56% accuracy. Considerable variability in performance highlights the need for careful validation and review before deploying NLP tools for clinical report labeling.

Related papers

• Auditable agentic AI for evidence-grounded thyroid ultrasound diagnosis and reporting [59.709162055944915]
  ThyroidXAgent is an agentic AI system that coordinates specialized diagnostic tools and stores their outputs as an auditable case-level evidence record.<n>It was evaluated on 28,458 non-overlapping test cases, including 8,721 cases from 35 centres in the private NHC-MISD-TUS cohort.<n>It achieved a mean Dice score of 87.21 percent for segmentation and a mean AUROC of 0.9466 for benign-malignant classification.
  arXiv  Detail & Related papers  (2026-08-12T21:02:59Z)
• Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset [0.0177677587528917]
  To evaluate whether large language model (LLM)-assisted label cleaning can identify label-report discordance in CT-RATE.<n>After report-level deduplication, 24,446 unique radiology reports were identified.<n>Overall agreement between GPT-5.4-derived and CT-RATE labels was 96.4%, with Cohen's kappa of 0.884.
  arXiv  Detail & Related papers  (2026-06-21T08:01:06Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• A DeepSeek-Powered AI System for Automated Chest Radiograph Interpretation in Clinical Practice [83.11942224668127]
  Janus-Pro-CXR (1B) is a chest X-ray interpretation system based on DeepSeek Janus-Pro model.<n>Our system outperforms state-of-the-art X-ray report generation models in automated report generation.
  arXiv  Detail & Related papers  (2025-12-23T13:26:13Z)
• Large Language Model-Based Uncertainty-Adjusted Label Extraction for Artificial Intelligence Model Development in Upper Extremity Radiography [1.1994929325702173]
  GPT-4o filled out structured templates by indicating imaging findings as present ("true"), absent ("false"), or "uncertain"<n>To assess the impact of label uncertainty, "uncertain" labels of the training and validation sets were automatically reassigned to "true" (inclusive) or "false"
  arXiv  Detail & Related papers  (2025-10-07T08:19:18Z)
• CLEAR: A Clinically-Grounded Tabular Framework for Radiology Report Evaluation [19.416198842242856]
  We introduce a Clinically-grounded framework with Expert-curated labels and Attribute-level comparison for Radiology report evaluation (CLEAR)<n>CLEAR examines whether a report can accurately identify the presence or absence of medical conditions.<n>To measure the clinical alignment of CLEAR, we collaborate with five board-certified radiologists to develop CLEAR-Bench, a dataset of 100 chest X-ray reports from MIMIC-CXR.
  arXiv  Detail & Related papers  (2025-05-22T07:32:12Z)
• CIRCA: comprehensible online system in support of chest X-rays-based COVID-19 diagnosis [37.41181188499616]
  Deep learning techniques can help in the faster detection of COVID-19 cases and monitoring of disease progression. Five different datasets were used to construct a representative dataset of 23 799 CXRs for model training. A U-Net-based model was developed to identify a clinically relevant region of the CXR.
  arXiv  Detail & Related papers  (2022-10-11T13:30:34Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• Supervised Machine Learning Algorithm for Detecting Consistency between Reported Findings and the Conclusions of Mammography Reports [66.89977257992568]
  Mammography reports document the diagnosis of patients' conditions. Many reports contain non-standard terms (non-BI-RADS descriptors) and incomplete statements. Our aim was to develop a tool to detect such discrepancies by comparing the reported conclusions to those that would be expected based on the reported radiology findings.
  arXiv  Detail & Related papers  (2022-02-28T08:59:04Z)
• Label-Assemble: Leveraging Multiple Datasets with Partial Labels [68.46767639240564]
  "Label-Assemble" aims to unleash the full potential of partial labels from an assembly of public datasets. We discovered that learning from negative examples facilitates both computer-aided disease diagnosis and detection.
  arXiv  Detail & Related papers  (2021-09-25T02:48:17Z)
• Quantification of pulmonary involvement in COVID-19 pneumonia by means of a cascade oftwo U-nets: training and assessment on multipledatasets using different annotation criteria [83.83783947027392]
  This study aims at exploiting Artificial intelligence (AI) for the identification, segmentation and quantification of COVID-19 pulmonary lesions. We developed an automated analysis pipeline, the LungQuant system, based on a cascade of two U-nets. The accuracy in predicting the CT-Severity Score (CT-SS) of the LungQuant system has been also evaluated.
  arXiv  Detail & Related papers  (2021-05-06T10:21:28Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Chest x-ray automated triage: a semiologic approach designed for clinical implementation, exploiting different types of labels through a combination of four Deep Learning architectures [83.48996461770017]
  This work presents a Deep Learning method based on the late fusion of different convolutional architectures. We built four training datasets combining images from public chest x-ray datasets and our institutional archive. We trained four different Deep Learning architectures and combined their outputs with a late fusion strategy, obtaining a unified tool.
  arXiv  Detail & Related papers  (2020-12-23T14:38:35Z)
• Auxiliary Diagnosing Coronary Stenosis Using Machine Learning [2.4100803794273]
  The four machine learning (ML) algorithms, i.e., Boosted Tree (BT), Decision Tree (DT), Logistic Regression (LR) and Random Forest (RF) are employed in this paper. The experimental results show that: RF performs better than other three algorithms, and the former algorithm classifies whether an individual has CS with an accuracy of 95.7%.
  arXiv  Detail & Related papers  (2020-07-16T04:56:57Z)
• Interpreting Chest X-rays via CNNs that Exploit Hierarchical Disease Dependencies and Uncertainty Labels [0.33598755777055367]
  We present a framework based on deep convolutional neural networks (CNNs) for diagnos-ing the presence of 14 common thoracic diseases and observations. The proposed method was also evaluated on an inde-pendent test set of the CheXpert competition, containing 500 CXR studies annotated by apanel of 5 experienced radiologists.
  arXiv  Detail & Related papers  (2020-05-25T11:07:53Z)
• CheXclusion: Fairness gaps in deep chest X-ray classifiers [4.656202572362684]
  We examine the extent to which state-of-the-art deep learning classifiers are biased with respect to protected attributes. We train convolution neural networks to predict 14 diagnostic labels in 3 prominent public chest X-ray datasets. We find that TPR disparities are not significantly correlated with a subgroup's proportional disease burden.
  arXiv  Detail & Related papers  (2020-02-14T22:08:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.