Zero-Shot ATC Coding with Large Language Models for Clinical Assessments

• URL: http://arxiv.org/abs/2412.07743v1
• Date: Tue, 10 Dec 2024 18:43:02 GMT
• Title: Zero-Shot ATC Coding with Large Language Models for Clinical Assessments
• Authors: Zijian Chen, John-Michael Gamble, Micaela Jantzi, John P. Hirdes, Jimmy Lin,
• Abstract summary: Manual assignment of Anatomical Therapeutic Chemical codes to prescription records is a significant bottleneck.<n>We develop a practical approach using locally deployable large language models (LLMs)<n>We evaluate our approach using GPT-4o as an accuracy ceiling and focus development on open-source Llama models suitable for privacy-sensitive deployment.
• Score: 40.72273945475456
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Manual assignment of Anatomical Therapeutic Chemical (ATC) codes to prescription records is a significant bottleneck in healthcare research and operations at Ontario Health and InterRAI Canada, requiring extensive expert time and effort. To automate this process while maintaining data privacy, we develop a practical approach using locally deployable large language models (LLMs). Inspired by recent advances in automatic International Classification of Diseases (ICD) coding, our method frames ATC coding as a hierarchical information extraction task, guiding LLMs through the ATC ontology level by level. We evaluate our approach using GPT-4o as an accuracy ceiling and focus development on open-source Llama models suitable for privacy-sensitive deployment. Testing across Health Canada drug product data, the RABBITS benchmark, and real clinical notes from Ontario Health, our method achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B. We investigate knowledge grounding through drug definitions, finding modest improvements in accuracy. Further, we show that fine-tuned Llama 3.1 8B matches zero-shot Llama 3.1 70B accuracy, suggesting that effective ATC coding is feasible with smaller models. Our results demonstrate the feasibility of automatic ATC coding in privacy-sensitive healthcare environments, providing a foundation for future deployments.

Related papers

• InformGen: An AI Copilot for Accurate and Compliant Clinical Research Consent Document Generation [22.52678425661723]
  We present InformGen, an LLM-driven copilot for accurate and compliant informed consent forms (ICFs) drafting. Experimental results demonstrate that InformGen achieves near 100% compliance with 18 core regulatory rules derived from FDA guidelines. When integrated with manual intervention, InformGen attains over 90% factual accuracy, significantly surpassing the vanilla GPT-4o model's 57%-82%.
  arXiv  Detail & Related papers  (2025-04-01T16:14:48Z)
• Practical Design and Benchmarking of Generative AI Applications for Surgical Billing and Coding [27.93881956637585]
  We present a strategy for developing generative AI tools for medical billing and coding. Our study shows that a small model that is fine-tuned on domain-specific data performs as well as the larger contemporary consumer models.
  arXiv  Detail & Related papers  (2025-01-07T17:11:12Z)
• Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Coding [44.01429184037945]
  We introduce ALIGN, a novel compositional LLM-based system for automated, zero-shot medical coding. We evaluate ALIGN on harmonizing medication terms into Anatomical Therapeutic Chemical (ATC) and medical history terms into Medical Dictionary for Regulatory Activities (MedDRA) codes.
  arXiv  Detail & Related papers  (2024-11-20T09:59:12Z)
• Novel Development of LLM Driven mCODE Data Model for Improved Clinical Trial Matching to Enable Standardization and Interoperability in Oncology Research [0.15346678870160887]
  Cancer costs reaching over $208 billion in 2023 alone. Traditional methods regarding clinical trial enrollment and clinical care in oncology are often manual, time-consuming, and lack a data-driven approach. This paper presents a novel framework to streamline standardization, interoperability, and exchange of cancer domains.
  arXiv  Detail & Related papers  (2024-10-18T17:31:35Z)
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs [54.05511925104712]
  We propose a simple, effective, and data-efficient method called Step-DPO. Step-DPO treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters.
  arXiv  Detail & Related papers  (2024-06-26T17:43:06Z)
• Cloud-based Federated Learning Framework for MRI Segmentation [0.10878040851637999]
  This paper proposes a novel framework tailored for brain tissue segmentation in rural healthcare facilities. The framework employs a deep reinforcement learning environment in tandem with a refinement model (RM) deployed locally at rural healthcare sites. We demonstrate the efficacy of our approach by training the network with a limited data set and observing a substantial performance enhancement.
  arXiv  Detail & Related papers  (2024-03-01T03:39:17Z)
• Automated clinical coding using off-the-shelf large language models [10.365958121087305]
  The task of assigning diagnostic ICD codes to patient hospital admissions is typically performed by expert human coders. Efforts towards automated ICD coding are dominated by supervised deep learning models. In this work, we leverage off-the-shelf pre-trained generative large language models to develop a practical solution.
  arXiv  Detail & Related papers  (2023-10-10T11:56:48Z)
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records [60.35217378132709]
  Large language models (LLMs) can follow natural language instructions with human-level fluency. evaluating LLMs on realistic text generation tasks for healthcare remains challenging. We introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data.
  arXiv  Detail & Related papers  (2023-08-27T12:24:39Z)
• Active learning for medical code assignment [55.99831806138029]
  We demonstrate the effectiveness of Active Learning (AL) in multi-label text classification in the clinical domain. We apply a set of well-known AL methods to help automatically assign ICD-9 codes on the MIMIC-III dataset. Our results show that the selection of informative instances provides satisfactory classification with a significantly reduced training set.
  arXiv  Detail & Related papers  (2021-04-12T18:11:17Z)
• ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed Quality Labeling Using Neural Networks [69.25956542388653]
  Deep learning (DL) algorithms are gaining weight in academic and industrial settings. We demonstrate DL can be successfully applied to low interpretative tasks by embedding ECG detection and delineation onto a segmentation framework. The model was trained using PhysioNet's QT database, comprised of 105 ambulatory ECG recordings.
  arXiv  Detail & Related papers  (2020-05-11T16:29:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.