Evaluating Generative AI as an Educational Tool for Radiology Resident Report Drafting

• URL: http://arxiv.org/abs/2511.02839v1
• Date: Mon, 22 Sep 2025 20:51:09 GMT
• Title: Evaluating Generative AI as an Educational Tool for Radiology Resident Report Drafting
• Abstract summary: This study evaluates a HIPAA-compliant GPT-4o system that delivers automated feedback on breast imaging reports drafted by residents in real clinical settings.<n>Three common error types were identified: (1) omission or addition of key findings, (2) incorrect use or omission of technical descriptors, and (3) final assessment inconsistent with findings.<n>GPT-4o showed strong agreement with attending consensus: 90.5%, 78.3%, and 90.4% across error types.
• Score: 1.5457333450799497
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Objective: Radiology residents require timely, personalized feedback to develop accurate image analysis and reporting skills. Increasing clinical workload often limits attendings' ability to provide guidance. This study evaluates a HIPAA-compliant GPT-4o system that delivers automated feedback on breast imaging reports drafted by residents in real clinical settings. Methods: We analyzed 5,000 resident-attending report pairs from routine practice at a multi-site U.S. health system. GPT-4o was prompted with clinical instructions to identify common errors and provide feedback. A reader study using 100 report pairs was conducted. Four attending radiologists and four residents independently reviewed each pair, determined whether predefined error types were present, and rated GPT-4o's feedback as helpful or not. Agreement between GPT and readers was assessed using percent match. Inter-reader reliability was measured with Krippendorff's alpha. Educational value was measured as the proportion of cases rated helpful. Results: Three common error types were identified: (1) omission or addition of key findings, (2) incorrect use or omission of technical descriptors, and (3) final assessment inconsistent with findings. GPT-4o showed strong agreement with attending consensus: 90.5%, 78.3%, and 90.4% across error types. Inter-reader reliability showed moderate variability (α = 0.767, 0.595, 0.567), and replacing a human reader with GPT-4o did not significantly affect agreement (Δ = -0.004 to 0.002). GPT's feedback was rated helpful in most cases: 89.8%, 83.0%, and 92.0%. Discussion: ChatGPT-4o can reliably identify key educational errors. It may serve as a scalable tool to support radiology education.

Related papers

• Evaluating the Safety of Deep Learning-Based Brain MRI Reconstruction [0.0]
  Deep learning can erase lesions or synthesize false tissue, but failures pixel-averaged metrics like PSNR and SSIM miss.<n>Following PRISMA 2020, we searched seven databases without date limits, including 263 studies (1995-2026)<n>Only 18 of 263 studies (6.8%) recorded both a fidelity metric and reader assessment on identical data, leaving the central surrogate unmeasured.
  arXiv  Detail & Related papers  (2026-08-28T08:08:14Z)
• Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs [0.0]
  We tested whether a structured evidence-sufficiency prompt reduces unsafe overconfident answers.<n>Blinded clinicians characterized the primary judge as a high-sensitivity (1.00), low-specificity (0.55) screen, not a calibrated scaffold gain.<n>Matched controls showed genuine behavior change, not judge circularity.
  arXiv  Detail & Related papers  (2026-07-20T15:53:02Z)
• Evaluation format, not model capability, drives triage failure in the assessment of consumer health AI [3.910019733981544]
  ChatGPT Health under-triages 51.6% of emergencies, concluding that consumer-facing AI triage poses safety risks.<n>We tested five frontier LLMs (GPT-5.2, Claude Sonnet 4.6, Claude Opus 4.6, Gemini 3 Flash, Gemini 3.1 Pro) on a 17-scenario partial replication bank.<n>The headline under-triage rate is highly contingent on evaluation format and should not be interpreted as a stable estimate of deployed triage behavior.
  arXiv  Detail & Related papers  (2026-03-12T00:58:22Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems [19.880569341968023]
  Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety.<n>We propose a retrieval-augmented multi-agent framework designed to automate the generation of instance-specific evaluation rubrics.
  arXiv  Detail & Related papers  (2026-01-21T16:40:41Z)
• Generating Natural-Language Surgical Feedback: From Structured Representation to Domain-Grounded Evaluation [66.7752700084159]
  High-quality feedback from a surgical trainer is pivotal for improving trainee performance and long-term skill acquisition.<n>We present a structure-aware pipeline that learns a surgical action ontology from real trainer-to-trainee transcripts.
  arXiv  Detail & Related papers  (2025-11-19T06:19:34Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• Mapping Patient-Perceived Physician Traits from Nationwide Online Reviews with LLMs [3.364244912862208]
  We present a large language model (LLM)-based pipeline that infers Big Five personality traits and five patient subjective judgments.<n>The analysis encompasses 4.1 million patient reviews of 226,999 U.S. physicians from an initial pool of one million.
  arXiv  Detail & Related papers  (2025-10-05T02:16:35Z)
• Benchmarking GPT-5 in Radiation Oncology: Measurable Gains, but Persistent Need for Expert Oversight [1.0471566053937098]
  GPT-5 is a large language model that has been specifically marketed towards oncology use.<n>On the TXIT benchmark, GPT-5 achieved a mean accuracy of 92.8%, outperforming GPT-4 (78.8%) and GPT-3.5 (62.1%)<n>In the vignette evaluation, GPT-5's treatment recommendations were rated highly for correctness (mean 3.24/4, 95% CI: 3.11-3.38) and comprehensiveness (3.59/4, 95% CI: 3.49-3.69)<n>While hallucinations were infrequent, the presence of substantive errors underscores that GPT-5-generated recommendations require rigorous expert oversight before clinical implementation
  arXiv  Detail & Related papers  (2025-08-29T16:55:25Z)
• Multi-Centre Validation of a Deep Learning Model for Scoliosis Assessment [0.0]
  We conducted a retrospective, multi centre evaluation of a fully automated deep learning software (Carebot AI Bones, Spine Measurement functionality; Carebot s.r.o.)<n>On 103 standing anteroposterior whole spine radiographs collected from ten hospitals.<n>Two musculoskeletal radiologists independently measured each study and served as reference readers.
  arXiv  Detail & Related papers  (2025-07-18T17:21:53Z)
• Interpretable Artificial Intelligence for Detecting Acute Heart Failure on Acute Chest CT Scans [2.2192473101240764]
  Chest CT scans are increasingly used in dyspneic patients where acute heart failure (AHF) is a key differential diagnosis.<n>We aim to develop an explainable AI model to detect radiological signs of AHF in chest CT with an accuracy comparable to thoracic radiologists.
  arXiv  Detail & Related papers  (2025-07-11T18:25:34Z)
• Evaluating GPT's Capability in Identifying Stages of Cognitive Impairment from Electronic Health Data [0.8777457069049611]
  This study evaluates an automated approach using zero-shot GPT-4o to determine stage of cognitive impairment in two different tasks.<n>We evaluated GPT-4o's ability to determine the global Clinical Dementia Rating (CDR) on specialist notes from 769 patients.<n>Second, we assessed GPT-4o's ability to differentiate between normal cognition, mild cognitive impairment (MCI), and dementia on all notes in a 3-year window from 860 Medicare patients.
  arXiv  Detail & Related papers  (2025-02-13T19:04:47Z)
• Impact of Large Language Model Assistance on Patients Reading Clinical Notes: A Mixed-Methods Study [46.5728291706842]
  We developed a patient-facing tool using large language models (LLMs) to make clinical notes more readable. We piloted the tool with clinical notes donated by patients with a history of breast cancer and synthetic notes from a clinician.
  arXiv  Detail & Related papers  (2024-01-17T23:14:52Z)
• A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis [87.25494411021066]
  GPT-4V's multimodal capability for medical image analysis is evaluated. It is found that GPT-4V excels in understanding medical images and generates high-quality radiology reports. It is found that its performance for medical visual grounding needs to be substantially improved.
  arXiv  Detail & Related papers  (2023-10-31T11:39:09Z)
• Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings [63.35165397320137]
  This study investigates the consistency of feedback ratings generated by OpenAI's GPT-4. The model rated responses to tasks within the Higher Education subject domain of macroeconomics in terms of their content and style.
  arXiv  Detail & Related papers  (2023-08-03T12:47:17Z)
• Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation [56.25869366777579]
  In recent years, machine learning models have rapidly become better at generating clinical consultation notes. We present an extensive human evaluation study where 5 clinicians listen to 57 mock consultations, write their own notes, post-edit a number of automatically generated notes, and extract all the errors. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore.
  arXiv  Detail & Related papers  (2022-04-01T14:04:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.