Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design

• URL: http://arxiv.org/abs/2604.17988v1
• Date: Mon, 20 Apr 2026 09:10:30 GMT
• Title: Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design
• Abstract summary: Off-the-shelf general-purpose LLMs offer superior support for pharmacoepidemiologic design compared to biomedical LLMs.<n>GPT-4o and DeepSeek-R1 paired with LTM prompting achieved the highest relevance and logic of justification scores.
• Score: 1.165082188315519
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: The potential of large language models (LLMs) to automate and support pharmacoepidemiologic study design is an emerging area of interest, yet their reliability remains insufficiently characterized. General-purpose LLMs often display inaccuracies, while the comparative performance of specialized biomedical LLMs in this domain remains unknown. Methods: This study evaluated general-purpose LLMs (GPT-4o and DeepSeek-R1) versus biomedically fine-tuned LLMs (QuantFactory/Bio-Medical-Llama-3-8B-GGUF and Irathernotsay/qwen2-1.5B-medical_qa-Finetune) using 46 protocols (2018-2024) from the HMA-EMA Catalogue and Sentinel System. Performance was assessed across relevance, logic of justification, and ontology-code agreement across multiple coding systems using Least-to-Most (LTM) and Active Prompting strategies. Results: GPT-4o and DeepSeek-R1 paired with LTM prompting achieved the highest relevance and logic of justification scores, with GPT-4o-LTM reaching a median relevance score of 4 in 8 of 9 questions for HMA-EMA protocols. Biomedical LLMs showed lower relevance overall and frequently generated insufficient justification. All LLMs demonstrated limited proficiency in ontology-code mapping, although LTM provided the most consistent improvements in reasoning stability. Conclusion: Off-the-shelf general-purpose LLMs currently offer superior support for pharmacoepidemiologic design compared to biomedical LLMs. Prompt strategy strongly influenced LLM performance.

Related papers

• Benchmarking Resource-Efficient LLMs for Research Topic Ontology Generation in the Biomedical Field [0.8633013637160062]
  We assess the performance of five small, open-source Large Language Models (LLMs) in identifying semantic relationships between biomedical concepts.<n>We analyse three adaptation strategies: standard prompting, Chain-of-Thought, and fine-tuning.<n>These results confirm that direct fine-tuning effectively exceeds the reasoning bottlenecks of smaller LLMs.
  arXiv  Detail & Related papers  (2026-07-20T12:48:41Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework [0.0]
  We present a retrieval-augmented generation (RAG) based medical QA system that combines domain-specific knowledge retrieval with open-source LLMs to answer medical questions.<n>We fine-tune two state-of-the-art open LLMs (LLaMA2 and Falcon) using Low-Rank Adaptation (LoRA) for efficient domain specialization.<n>Our fine-tuned LLaMA2 model achieves 71.8% accuracy on PubMedQA, substantially improving over the 55.4% zero-shot baseline.
  arXiv  Detail & Related papers  (2025-12-05T16:38:47Z)
• BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text [14.409097921305134]
  BRIDGE is a comprehensive benchmark comprising 87 tasks sourced from real-world clinical data sources across nine languages.<n>It covers eight major task types spanning the entire continuum of patient care across six clinical stages and 20 representative applications.<n>Our results reveal substantial performance variation across model sizes, languages, natural language processing tasks, and clinical specialties.
  arXiv  Detail & Related papers  (2025-04-28T04:13:18Z)
• LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition? [59.81732629438753]
  We propose LLaVA-RadZ, a simple yet effective framework for zero-shot medical disease recognition via utilizing the existing MLLM features.<n>Specifically, we design an end-to-end training strategy, termed Decoding-Side Feature Alignment Training (DFAT) to take advantage of the characteristics of the MLLM decoder architecture.<n>We also introduce a Domain Knowledge Anchoring Module (DKAM) to exploit the intrinsic medical knowledge of large models.
  arXiv  Detail & Related papers  (2025-03-10T16:05:40Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI [67.09501109871351]
  Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals. GMAI-MMBench is the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format.
  arXiv  Detail & Related papers  (2024-08-06T17:59:21Z)
• Lightweight Large Language Model for Medication Enquiry: Med-Pal [2.3095351248532268]
  Large Language Models (LLMs) have emerged as a potential solution to assist digital health development with patient education. We trained and validated Med-Pal, a medication domain-specific LLM-chatbot fine-tuned with a fine-grained and expert curated dataset.
  arXiv  Detail & Related papers  (2024-07-02T03:32:39Z)
• LLMs in Biomedicine: A study on clinical Named Entity Recognition [42.71263594812782]
  Large Language Models (LLMs) demonstrate remarkable versatility in various NLP tasks. This paper investigates strategies to enhance their performance for the NER task. Our proposed method, DiRAG, can boost the zero-shot F1 score of LLMs for biomedical NER.
  arXiv  Detail & Related papers  (2024-04-10T22:26:26Z)
• RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain [0.37282630026096586]
  We introduce the Reliability AssesMent for Biomedical LLM Assistants framework. We evaluate whether four state-of-the-art foundation LLMs can serve as reliable assistants in the biomedical domain.
  arXiv  Detail & Related papers  (2024-03-21T17:30:59Z)
• Large Language Model Distilling Medication Recommendation Model [58.94186280631342]
  We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs)<n>Our research aims to transform existing medication recommendation methodologies using LLMs.<n>To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
  arXiv  Detail & Related papers  (2024-02-05T08:25:22Z)
• Quantifying Self-diagnostic Atomic Knowledge in Chinese Medical Foundation Model: A Computational Analysis [55.742339781494046]
  Foundation Models (FMs) have the potential to revolutionize the way users self-diagnose through search engines by offering direct and efficient suggestions. Recent studies primarily focused on the quality of FMs evaluated by GPT-4 or their ability to pass medical exams. No studies have quantified the extent of self-diagnostic atomic knowledge stored in FMs' memory.
  arXiv  Detail & Related papers  (2023-10-18T05:42:22Z)
• Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization [66.08074487429477]
  We investigate the stability and reliability of large language models (LLMs) as automatic evaluators for abstractive summarization. We find that while ChatGPT and GPT-4 outperform the commonly used automatic metrics, they are not ready as human replacements.
  arXiv  Detail & Related papers  (2023-05-22T14:58:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.