Large Language Model-Based Uncertainty-Adjusted Label Extraction for Artificial Intelligence Model Development in Upper Extremity Radiography

• URL: http://arxiv.org/abs/2510.05664v1
• Date: Tue, 07 Oct 2025 08:19:18 GMT
• Title: Large Language Model-Based Uncertainty-Adjusted Label Extraction for Artificial Intelligence Model Development in Upper Extremity Radiography
• Abstract summary: GPT-4o filled out structured templates by indicating imaging findings as present ("true"), absent ("false"), or "uncertain"<n>To assess the impact of label uncertainty, "uncertain" labels of the training and validation sets were automatically reassigned to "true" (inclusive) or "false"
• Score: 1.1994929325702173
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Objectives: To evaluate GPT-4o's ability to extract diagnostic labels (with uncertainty) from free-text radiology reports and to test how these labels affect multi-label image classification of musculoskeletal radiographs. Methods: This retrospective study included radiography series of the clavicle (n=1,170), elbow (n=3,755), and thumb (n=1,978). After anonymization, GPT-4o filled out structured templates by indicating imaging findings as present ("true"), absent ("false"), or "uncertain." To assess the impact of label uncertainty, "uncertain" labels of the training and validation sets were automatically reassigned to "true" (inclusive) or "false" (exclusive). Label-image-pairs were used for multi-label classification using ResNet50. Label extraction accuracy was manually verified on internal (clavicle: n=233, elbow: n=745, thumb: n=393) and external test sets (n=300 for each). Performance was assessed using macro-averaged receiver operating characteristic (ROC) area under the curve (AUC), precision recall curves, sensitivity, specificity, and accuracy. AUCs were compared with the DeLong test. Results: Automatic extraction was correct in 98.6% (60,618 of 61,488) of labels in the test sets. Across anatomic regions, label-based model training yielded competitive performance measured by macro-averaged AUC values for inclusive (e.g., elbow: AUC=0.80 [range, 0.62-0.87]) and exclusive models (elbow: AUC=0.80 [range, 0.61-0.88]). Models generalized well on external datasets (elbow [inclusive]: AUC=0.79 [range, 0.61-0.87]; elbow [exclusive]: AUC=0.79 [range, 0.63-0.89]). No significant differences were observed across labeling strategies or datasets (p>=0.15). Conclusion: GPT-4o extracted labels from radiologic reports to train competitive multi-label classification models with high accuracy. Detected uncertainty in the radiologic reports did not influence the performance of these models.

Related papers

• TRIAGE: Risk-Controlled Pseudo-Label Admission for Annotation-Efficient Semi-Supervised Retinal OCT Classification [0.6213771671016098]
  This paper proposes TRIAGE, a risk-controlled semi-supervised framework for OCT scans classification.<n>On the dataset from Noor Eye Hospital (16,822 B-scans, 161 patients, and 554 volumes), TRIAGE demonstrates 89.66% scan-level accuracy, 0.8805 macro-F1, 0.9641 macro-AUC, and an 8.34% under-grading rate.
  arXiv  Detail & Related papers  (2026-08-14T14:06:48Z)
• Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset [0.0177677587528917]
  To evaluate whether large language model (LLM)-assisted label cleaning can identify label-report discordance in CT-RATE.<n>After report-level deduplication, 24,446 unique radiology reports were identified.<n>Overall agreement between GPT-5.4-derived and CT-RATE labels was 96.4%, with Cohen's kappa of 0.884.
  arXiv  Detail & Related papers  (2026-06-21T08:01:06Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Conformal Prediction for Risk-Controlled Medical Entity Extraction Across Clinical Domains [2.744879848686032]
  We present a conformal prediction framework that provides finite-sample coverage guarantees for medical entity extraction.<n>First, we extract structured entities from 1,000 FDA drug labels across eight sections using GPT-4.1, verified via FactScore-based atomic statement evaluation.<n>Second, we extract radiological entities from MIMIC-CXR reports using the RadGraph schema with GPT-4.1 and Llama-4-Maverick, evaluated against physician annotations.
  arXiv  Detail & Related papers  (2026-03-01T05:12:25Z)
• Performance of a Deep Learning-Based Segmentation Model for Pancreatic Tumors on Public Endoscopic Ultrasound Datasets [2.925528117330222]
  Pancreatic cancer is one of the most aggressive cancers, with poor survival rates.<n>This study evaluates a Vision Transformer-based deep learning segmentation model for pancreatic tumors.
  arXiv  Detail & Related papers  (2026-01-09T16:48:50Z)
• Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset [0.0]
  This study compares four commercial clinical NLP systems for entity extraction and assertion detection.<n>CheXpert and CheXbert, two dedicated chest radiograph report labelers, were evaluated on the same task.<n>We analyzed 95,008 pediatric CXR reports from a large academic pediatric hospital.
  arXiv  Detail & Related papers  (2025-05-29T03:16:18Z)
• Automated Spinal MRI Labelling from Reports Using a Large Language Model [45.348320669329205]
  We propose a pipeline to automate the extraction of labels from radiology reports using large language models. Our method equals or surpasses GPT-4 on a held-out set of reports. We show that the extracted labels can be used to train imaging models to classify the identified conditions in the accompanying MR scans.
  arXiv  Detail & Related papers  (2024-10-22T17:54:07Z)
• Improving Image Classification of Knee Radiographs: An Automated Image Labeling Approach [0.3258500021481664]
  The purpose of our study was to develop an automated labeling approach that improves the image classification to distinguish normal knee images with abnormalities or prior. The automated labeler was trained on a small set of labeled data to automatically label a much larger set of unlabeled data. Our findings demonstrated that the proposed automated labeling approach significantly improves the performance of image classification for knee diagnosis.
  arXiv  Detail & Related papers  (2023-09-06T03:26:24Z)
• Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification [52.77024349608834]
  To investigate chest radiograph (CXR) classification performance of vision transformers (ViT) and interpretability of attention-based saliency. ViTs were fine-tuned for lung disease classification using four public data sets: CheXpert, Chest X-Ray 14, MIMIC CXR, and VinBigData. ViTs had comparable CXR classification AUCs compared with state-of-the-art CNNs.
  arXiv  Detail & Related papers  (2023-03-03T12:05:41Z)
• Learning to diagnose common thorax diseases on chest radiographs from radiology reports in Vietnamese [0.33598755777055367]
  We propose a data collecting and annotation pipeline that extracts information from Vietnamese radiology reports to provide accurate labels for chest X-ray (CXR) images. This can benefit Vietnamese radiologists and clinicians by annotating data that closely match their endemic diagnosis categories which may vary from country to country.
  arXiv  Detail & Related papers  (2022-09-11T06:06:03Z)
• Less is More: Adaptive Curriculum Learning for Thyroid Nodule Diagnosis [50.231954872304314]
  We propose an Adaptive Curriculum Learning framework, which adaptively discovers and discards the samples with inconsistent labels. We also contribute TNCD: a Thyroid Nodule Classification dataset.
  arXiv  Detail & Related papers  (2022-07-02T11:50:02Z)
• Report-Guided Automatic Lesion Annotation for Deep Learning-Based Prostate Cancer Detection in bpMRI [0.0]
  Automatic annotation based on clinical reports could overcome the manual labelling bottleneck. We generated clinically significant prostate cancer (csPCa) annotations. Augmenting the training set with automatically labelled exams improved patient-based diagnostic area.
  arXiv  Detail & Related papers  (2021-12-09T15:35:32Z)
• Semi-supervised learning for generalizable intracranial hemorrhage detection and segmentation [0.0]
  We develop and evaluate a semisupervised learning model for intracranial hemorrhage detection and segmentation on an outofdistribution head CT evaluation set. An initial "teacher" deep learning model was trained on 457 pixel-labeled head CT scans collected from one US institution from 2010-2017. A second "student" model was trained on this combined pixel-labeled and pseudo-labeled dataset.
  arXiv  Detail & Related papers  (2021-05-03T00:14:43Z)
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning [53.1047775185362]
  Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process.
  arXiv  Detail & Related papers  (2021-01-15T23:29:57Z)
• Chest x-ray automated triage: a semiologic approach designed for clinical implementation, exploiting different types of labels through a combination of four Deep Learning architectures [83.48996461770017]
  This work presents a Deep Learning method based on the late fusion of different convolutional architectures. We built four training datasets combining images from public chest x-ray datasets and our institutional archive. We trained four different Deep Learning architectures and combined their outputs with a late fusion strategy, obtaining a unified tool.
  arXiv  Detail & Related papers  (2020-12-23T14:38:35Z)
• Semi-Supervised Speech Recognition via Graph-based Temporal Classification [59.58318952000571]
  Semi-supervised learning has demonstrated promising results in automatic speech recognition by self-training. The effectiveness of this approach largely relies on the pseudo-label accuracy. Alternative ASR hypotheses of an N-best list can provide more accurate labels for an unlabeled speech utterance.
  arXiv  Detail & Related papers  (2020-10-29T14:56:56Z)
• Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes [64.21642241351857]
  We curated and analyzed a chest computed tomography (CT) data set of 36,316 volumes from 19,993 unique patients. We developed a rule-based method for automatically extracting abnormality labels from free-text radiology reports. We also developed a model for multi-organ, multi-disease classification of chest CT volumes.
  arXiv  Detail & Related papers  (2020-02-12T00:59:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.