Large language models management of medications: three performance analyses

• URL: http://arxiv.org/abs/2509.22926v2
• Date: Tue, 14 Oct 2025 15:32:00 GMT
• Title: Large language models management of medications: three performance analyses
• Abstract summary: The performance of GPT 4o was tested for three medication management tasks.<n>For the first task of drug-formulation matching, GPT-4o had 49% accuracy for generic medications being matched to all available formulations.<n>For the second task of drug-drug interaction identification, the accuracy was 54.7% for identifying the DDI pair.
• Score: 9.12176336117872
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Purpose: Large language models (LLMs) have proven performance for certain diagnostic tasks, however limited studies have evaluated their consistency in recommending appropriate medication regimens for a given diagnosis. Medication management is a complex task that requires synthesis of drug formulation and complete order instructions for safe use. Here, the performance of GPT 4o, an LLM available with ChatGPT, was tested for three medication management tasks. Methods: GPT-4o performance was tested using three medication tasks: identifying available formulations for a given generic drug name, identifying drug-drug interactions (DDI) for a given medication regimen, and preparing a medication order for a given generic drug name. For each experiment, the models raw text response was captured exactly as returned and evaluated using clinician evaluation in addition to standard LLM metrics, including Term Frequency-Inverse Document Frequency (TF IDF) vectors, normalized Levenshtein similarity, and Recall-Oriented Understudy for Gisting Evaluation (ROUGE 1/ROUGE L F1) between each response and its reference string. Results: For the first task of drug-formulation matching, GPT-4o had 49% accuracy for generic medications being matched to all available formulations, with an average of 1.23 omissions per medication and 1.14 hallucinations per medication. For the second task of drug-drug interaction identification, the accuracy was 54.7% for identifying the DDI pair. For the third task, GPT-4o generated order sentences containing no medication or abbreviation errors in 65.8% of cases. Conclusions: Model performance for basic medication tasks was consistently poor. This evaluation highlights the need for domain-specific training through clinician-annotated datasets and a comprehensive evaluation framework for benchmarking performance.

Related papers

• GRAIN: Molecules Are Not the Right Granularity -- Active-Ingredient Modeling for Safe Medication Recommendation [2.26215287738502]
  Medication recommendation from electronic health records must balance predictive accuracy against the risk of adverse drug-drug interactions (DDIs) under polypharmacy.<n>We argue that the active ingredient is the missing granularity, and introduce GRAIN, a medication recommendation framework built around it.<n>GRAIN encodes longitudinal patient trajectories (diagnoses, procedures, past medications) with a selective state space backbone that handles long, irregular visit sequences in linear time.
  arXiv  Detail & Related papers  (2026-07-30T18:13:30Z)
• TxBench-PP: Analyzing AI Agent Performance on Small-Molecule Preclinical Pharmacology [0.0]
  Artificial intelligence (AI) agents promise to accelerate drug discovery by compressing interpretation and decision-making loops.<n>We introduce TherapeuticsBench Preclinical Pharmacology (TxBench-PP), a verifiable benchmark for small-molecule preclinical pharmacology.<n>TxBench-PP tests whether agents can recover accurate conclusions from real-world assay data rather than memorized facts from literature.
  arXiv  Detail & Related papers  (2026-06-17T16:23:45Z)
• PACE-RAG: Patient-Aware Contextual and Evidence-based Policy RAG for Clinical Drug Recommendation [40.19067516813213]
  PACE-RAG (Patient-Aware Contextual and Evidence-based Policy RAG) is a novel framework designed to synthesize individual patient context with the prescribing tendencies of similar cases.<n>By analyzing treatment patterns tailored to specific clinical signals, PACE-RAG identifies optimal prescriptions and generates an explainable clinical summary.
  arXiv  Detail & Related papers  (2026-03-18T04:40:53Z)
• Retrieval Augmented Large Language Model System for Comprehensive Drug Contraindications [0.0]
  The versatility of large language models (LLMs) has been explored across various sectors, but their application in healthcare poses challenges.<n>This study enhances the capability of LLMs to address contraindications effectively by implementing a Retrieval Augmented Generation (RAG) pipeline.
  arXiv  Detail & Related papers  (2025-08-08T09:09:03Z)
• Validating Pharmacogenomics Generative Artificial Intelligence Query Prompts Using Retrieval-Augmented Generation (RAG) [0.0]
  This study evaluated Sherpa Rx, an artificial intelligence tool leveraging large language models and retrieval-augmented generation (RAG) for pharmacogenomics.<n>Sherpa Rx integrated Clinical Pharmacogenetics Implementation Consortium (CPIC) guidelines with Pharmacogenomics Knowledgebase ( PharmGKB) data to generate contextually relevant responses.<n>A dataset (N=260 queries) spanning 26 CPIC guidelines was used to evaluate drug-gene interactions, dosing recommendations, and therapeutic implications.
  arXiv  Detail & Related papers  (2025-07-29T02:43:35Z)
• Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening [37.69303106863453]
  Large language models (LLMs) can simulate clinical reasoning based on natural language prompts, but their utility in ophthalmology is largely unexplored.<n>This study evaluated GPT-4's ability to interpret structured textual descriptions of retinal fundus photographs.<n>We conducted a retrospective diagnostic validation study using 300 annotated fundus images.
  arXiv  Detail & Related papers  (2025-07-02T01:35:59Z)
• Scalable Medication Extraction and Discontinuation Identification from Electronic Health Records Using Large Language Models [8.201290223465692]
  We evaluate advanced open-sourced and proprietary large language models (LLMs) in extracting medications and classifying their medication status from EHR notes.<n>GPT-4o consistently achieved the highest average F1 scores in all tasks under zero-shot setting.<n>Open-sourced models followed closely, Llama-3.1-70B-Instruct achieved the highest performance in medication status classification.
  arXiv  Detail & Related papers  (2025-06-10T22:00:32Z)
• Do "New Snow Tablets" Contain Snow? Large Language Models Over-Rely on Names to Identify Ingredients of Chinese Drugs [79.00288739947406]
  Traditional Chinese Medicine (TCM) has seen increasing adoption in healthcare, with specialized Large Language Models (LLMs) emerging to support clinical applications.<n>A fundamental requirement for these models is accurate identification of TCM drug ingredients.<n>Our systematic analysis reveals consistent failure patterns: models often interpret drug names literally, overuse common herbs regardless of relevance, and exhibit erratic behaviors when faced with unfamiliar formulations.
  arXiv  Detail & Related papers  (2025-04-03T17:43:45Z)
• MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models [82.30696225661615]
  We introduce MedHallu, the first benchmark specifically designed for medical hallucination detection.<n>We show that state-of-the-art LLMs, including GPT-4o, Llama-3.1, and medically fine-tuned UltraMedical, struggle with this binary hallucination detection task.<n>Using bidirectional entailment clustering, we show that harder-to-detect hallucinations are semantically closer to ground truth.
  arXiv  Detail & Related papers  (2025-02-20T06:33:23Z)
• Natural Language-Assisted Multi-modal Medication Recommendation [97.07805345563348]
  We introduce the Natural Language-Assisted Multi-modal Medication Recommendation(NLA-MMR)<n>The NLA-MMR is a multi-modal alignment framework designed to learn knowledge from the patient view and medication view jointly.<n>In this vein, we employ pretrained language models(PLMs) to extract in-domain knowledge regarding patients and medications.
  arXiv  Detail & Related papers  (2025-01-13T09:51:50Z)
• Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Coding [44.01429184037945]
  We introduce ALIGN, a novel compositional LLM-based system for automated, zero-shot medical coding.<n>We evaluate ALIGN on harmonizing medication terms into Anatomical Therapeutic Chemical (ATC) and medical history terms into Medical Dictionary for Regulatory Activities (MedDRA) codes.
  arXiv  Detail & Related papers  (2024-11-20T09:59:12Z)
• Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes [20.651573628726148]
  This study assesses the impact of lab test results on differential diagnoses made by large language models (LLMs) LLMs GPT-4, GPT-3.5, Llama-2-70b, Claude-2, and Mixtral-8x7B were tested to generate Top 10, Top 5, and Top 1 DDx with and without lab data. GPT-4 performed best, achieving 55% accuracy for Top 1 diagnoses and 60% for Top 10 with lab data, with lenient accuracy up to 80%. Lab tests, including liver function, metabolic/toxicology panels, and serology/immune tests, were generally interpreted correctly by
  arXiv  Detail & Related papers  (2024-11-01T02:48:32Z)
• A randomized simulation trial evaluating ABiMed, a clinical decision support system for medication reviews and polypharmacy management [3.8243906257653504]
  We designed ABiMed, a clinical decision support system for medication reviews, based on the implementation of the STOPP/START v2 guidelines. We evaluated ABiMed with 39 community pharmacists during a randomized simulation trial.
  arXiv  Detail & Related papers  (2024-09-03T13:50:59Z)
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records [60.35217378132709]
  Large language models (LLMs) can follow natural language instructions with human-level fluency. evaluating LLMs on realistic text generation tasks for healthcare remains challenging. We introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data.
  arXiv  Detail & Related papers  (2023-08-27T12:24:39Z)
• Improving Large Language Models for Clinical Named Entity Recognition via Prompt Engineering [20.534197056683695]
  This study quantifies the capabilities of GPT-3.5 and GPT-4 for clinical named entity recognition (NER) tasks. We developed a task-specific prompt framework that includes baseline prompts, annotation guideline-based prompts, error analysis-based instructions, and annotated samples. We assessed each prompt's effectiveness and compared the models to BioClinicalBERT.
  arXiv  Detail & Related papers  (2023-03-29T02:46:18Z)
• Neural Bandits for Data Mining: Searching for Dangerous Polypharmacy [63.135687276599114]
  Some polypharmacies, deemed inappropriate, may be associated with adverse health outcomes such as death or hospitalization. We propose the OptimNeuralTS strategy to efficiently mine claims datasets and build a predictive model of the association between drug combinations and health outcomes. Our method can detect up to 72% of PIPs while maintaining an average precision score of 99% using 30 000 time steps.
  arXiv  Detail & Related papers  (2022-12-10T03:43:23Z)
• Knowledge-Driven New Drug Recommendation [88.35607943144261]
  We develop a drug-dependent multi-phenotype few-shot learner to bridge the gap between existing and new drugs. EDGE eliminates the false-negative supervision signal using an external drug-disease knowledge base. Results show that EDGE achieves 7.3% improvement on the ROC-AUC score over the best baseline.
  arXiv  Detail & Related papers  (2022-10-11T16:07:52Z)
• Multi-View Substructure Learning for Drug-Drug Interaction Prediction [69.34322811160912]
  We propose a novel multi- view drug substructure network for DDI prediction (MSN-DDI) MSN-DDI learns chemical substructures from both the representations of the single drug (intra-view) and the drug pair (inter-view) simultaneously and utilizes the substructures to update the drug representation iteratively. Comprehensive evaluations demonstrate that MSN-DDI has almost solved DDI prediction for existing drugs by achieving a relatively improved accuracy of 19.32% and an over 99% accuracy under the transductive setting.
  arXiv  Detail & Related papers  (2022-03-28T05:44:29Z)
• SafeDrug: Dual Molecular Graph Encoders for Safe Drug Recommendations [59.590084937600764]
  We propose a DDI-controllable drug recommendation model named SafeDrug to leverage drugs' molecule structures and model DDIs explicitly. On a benchmark dataset, our SafeDrug is relatively shown to reduce DDI by 19.43% and improves 2.88% on Jaccard similarity between recommended and actually prescribed drug combinations over previous approaches.
  arXiv  Detail & Related papers  (2021-05-05T00:20:48Z)
• Drug Repurposing for COVID-19 via Knowledge Graph Completion [6.705100803382272]
  We propose a novel, integrative, and neural network-based literature-based discovery (LBD) approach to identify drug candidates. Our approach relies on semantic triples extracted using SemRep. Five SOTA, neural knowledge graph completion algorithms were used to predict drug repurposing candidates.
  arXiv  Detail & Related papers  (2020-10-19T15:30:51Z)
• Two Step Joint Model for Drug Drug Interaction Extraction [82.49278654043577]
  Drug-Drug Interaction (DDI) Extraction from Drug Labels challenge of Text Analysis Conference (TAC) 2018. We propose a two step joint model to detect DDI and it's related mentions jointly. A sequence tagging system (CNN-GRU encoder-decoder) finds precipitants first and search its fine-grained Trigger and determine the DDI for each precipitant in the second step.
  arXiv  Detail & Related papers  (2020-08-28T15:30:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.