Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset

• URL: http://arxiv.org/abs/2606.22382v1
• Date: Sun, 21 Jun 2026 08:01:06 GMT
• Title: Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset
• Abstract summary: To evaluate whether large language model (LLM)-assisted label cleaning can identify label-report discordance in CT-RATE.<n>After report-level deduplication, 24,446 unique radiology reports were identified.<n>Overall agreement between GPT-5.4-derived and CT-RATE labels was 96.4%, with Cohen's kappa of 0.884.
• Score: 0.0177677587528917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Purpose: To evaluate whether large language model (LLM)-assisted label cleaning can identify label-report discordance in CT-RATE, a large-scale public chest CT dataset. Materials and Methods: After report-level deduplication, 24,446 unique radiology reports were identified. Twelve reports were excluded from the primary GPT-5.4 analysis because of Microsoft Azure AI Foundry content-safety filtering, leaving 24,434 reports and 439,812 label instances across 18 abnormality categories. GPT-5.4-derived binary labels were generated from report text using structured JSON output and compared with existing CT-RATE labels. Discordant instances were adjudicated by radiologists. In addition, 100 randomly sampled reports were manually annotated to compare CT-RATE labels, individual LLM-derived labels, and multi-LLM majority-vote labels against radiologist-annotated reference labels. Results: Overall agreement between GPT-5.4-derived and CT-RATE labels was 96.4%, with Cohen's kappa of 0.884. Lymphadenopathy showed the lowest agreement and kappa. In discordance review, radiologist adjudication supported GPT-5.4-derived labels in 72 of 97 (74.2%) general discordant instances and 91 of 99 (91.9%) targeted lymphadenopathy discordant instances. Against radiologist-annotated reference labels, multi-LLM majority-vote labels achieved the highest label-macro-averaged F1 score and Cohen's kappa. Conclusion: LLM-assisted label cleaning identified clinically meaningful label-report discordance in CT-RATE and may support scalable quality improvement of public imaging datasets. The cleaned dataset will be made publicly available to support future research.

Related papers

• Reconfigurable Radiology Labels Without Relabeling [4.19448785173794]
  We release a pipeline that converts free-text reports into multi-label matrices.<n>We show that 43% of CXR studies contain at least one finding outside CheXpert-14.
  arXiv  Detail & Related papers  (2026-07-06T23:19:20Z)
• RSNA Large Language Model Benchmark Dataset for Chest Radiographs of Cardiothoracic Disease: Radiologist Evaluation and Validation Enhanced by AI Labels (REVEAL-CXR) [27.699008063855146]
  GPT-4o extracted abnormal findings from 13,735 chest radiographs and their corresponding reports from the MIDRC.<n>1,000 were sampled on the basis of the AI-suggested benchmark labels for expert review.<n>17 chest radiologists participated, and they marked "Agree all", "Agree mostly" or "Disagree" to indicate their assessment of the correctness of the LLM suggested labels.
  arXiv  Detail & Related papers  (2026-01-21T16:04:01Z)
• Large Language Model-Based Uncertainty-Adjusted Label Extraction for Artificial Intelligence Model Development in Upper Extremity Radiography [1.1994929325702173]
  GPT-4o filled out structured templates by indicating imaging findings as present ("true"), absent ("false"), or "uncertain"<n>To assess the impact of label uncertainty, "uncertain" labels of the training and validation sets were automatically reassigned to "true" (inclusive) or "false"
  arXiv  Detail & Related papers  (2025-10-07T08:19:18Z)
• Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset [0.0]
  This study compares four commercial clinical NLP systems for entity extraction and assertion detection.<n>CheXpert and CheXbert, two dedicated chest radiograph report labelers, were evaluated on the same task.<n>We analyzed 95,008 pediatric CXR reports from a large academic pediatric hospital.
  arXiv  Detail & Related papers  (2025-05-29T03:16:18Z)
• Label Distribution Learning with Biased Annotations by Learning Multi-Label Representation [120.97262070068224]
  Multi-label learning (MLL) has gained attention for its ability to represent real-world data.<n>Label Distribution Learning (LDL) faces challenges in collecting accurate label distributions.
  arXiv  Detail & Related papers  (2025-02-03T09:04:03Z)
• MIMIC-IV-Ext-PE: Using a large language model to predict pulmonary embolism phenotype in the MIMIC-IV dataset [0.0]
  Pulmonary embolism is a leading cause of preventable in-hospital mortality. There are few large publicly available datasets that contain PE labels for research. We extracted all available radiology reports ofCTPA scans and two physicians manually labeled the results as PE positive (acute PE) or PE negative. We applied a previously finetuned Bio_ClinicalBERT transformer language model, VTE-BERT, to extract labels automatically.
  arXiv  Detail & Related papers  (2024-10-29T19:28:44Z)
• WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinoma [51.50991881342181]
  This challenge includes 10,091 patch-level annotations and over 130 million labeled pixels. First place team achieved mIoU of 0.8413 (tumor: 0.8389, stroma: 0.7931, normal: 0.8919)
  arXiv  Detail & Related papers  (2022-04-13T15:27:05Z)
• Acknowledging the Unknown for Multi-label Learning with Single Positive Labels [65.5889334964149]
  Traditionally, all unannotated labels are assumed as negative labels in single positive multi-label learning (SPML) We propose entropy-maximization (EM) loss to maximize the entropy of predicted probabilities for all unannotated labels. Considering the positive-negative label imbalance of unannotated labels, we propose asymmetric pseudo-labeling (APL) with asymmetric-tolerance strategies and a self-paced procedure to provide more precise supervision.
  arXiv  Detail & Related papers  (2022-03-30T11:43:59Z)
• Supervised Machine Learning Algorithm for Detecting Consistency between Reported Findings and the Conclusions of Mammography Reports [66.89977257992568]
  Mammography reports document the diagnosis of patients' conditions. Many reports contain non-standard terms (non-BI-RADS descriptors) and incomplete statements. Our aim was to develop a tool to detect such discrepancies by comparing the reported conclusions to those that would be expected based on the reported radiology findings.
  arXiv  Detail & Related papers  (2022-02-28T08:59:04Z)
• Label-Assemble: Leveraging Multiple Datasets with Partial Labels [68.46767639240564]
  "Label-Assemble" aims to unleash the full potential of partial labels from an assembly of public datasets. We discovered that learning from negative examples facilitates both computer-aided disease diagnosis and detection.
  arXiv  Detail & Related papers  (2021-09-25T02:48:17Z)
• VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labels [4.865330207715854]
  We show that radiologists labeling reports significantly disagree with radiologists labeling chest X-ray images. We develop and evaluate methods to produce labels from radiology reports that have better agreement with radiologists labeling images.
  arXiv  Detail & Related papers  (2021-02-23T03:02:36Z)
• Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes [64.21642241351857]
  We curated and analyzed a chest computed tomography (CT) data set of 36,316 volumes from 19,993 unique patients. We developed a rule-based method for automatically extracting abnormality labels from free-text radiology reports. We also developed a model for multi-organ, multi-disease classification of chest CT volumes.
  arXiv  Detail & Related papers  (2020-02-12T00:59:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.