AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients

• URL: http://arxiv.org/abs/2502.15944v1
• Date: Fri, 21 Feb 2025 21:17:37 GMT
• Title: AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients
• Authors: Sean Wu, Michael Koo, Fabien Scalzo, Ira Kurtz,
• Abstract summary: Large language models (LLMs) have demonstrated increasingly sophisticated performance in medical and other fields of knowledge.<n>Recent prompt engineering, instead of fine-tuning, has shown potential to boost the performance of general foundation models.<n>We present AutoMedPrompt, which explores the use of textual gradients to elicit medically relevant reasoning.
• Score: 0.3636228980200798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated increasingly sophisticated performance in medical and other fields of knowledge. Traditional methods of creating specialist LLMs require extensive fine-tuning and training of models on large datasets. Recently, prompt engineering, instead of fine-tuning, has shown potential to boost the performance of general foundation models. However, prompting methods such as chain-of-thought (CoT) may not be suitable for all subspecialty, and k-shot approaches may introduce irrelevant tokens into the context space. We present AutoMedPrompt, which explores the use of textual gradients to elicit medically relevant reasoning through system prompt optimization. AutoMedPrompt leverages TextGrad's automatic differentiation via text to improve the ability of general foundation LLMs. We evaluated AutoMedPrompt on Llama 3, an open-source LLM, using several QA benchmarks, including MedQA, PubMedQA, and the nephrology subspecialty-specific NephSAP. Our results show that prompting with textual gradients outperforms previous methods on open-source LLMs and surpasses proprietary models such as GPT-4, Claude 3 Opus, and Med-PaLM 2. AutoMedPrompt sets a new state-of-the-art (SOTA) performance on PubMedQA with an accuracy of 82.6$\%$, while also outperforming previous prompting strategies on open-sourced models for MedQA (77.7$\%$) and NephSAP (63.8$\%$).

Related papers

• BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text [10.071956824618418]
  Large language models (LLMs) hold great promise for medical applications and are evolving rapidly. Most existing benchmarks rely on medical exam-style questions or PubMed-derived text. We present BRIDGE, a comprehensive benchmark comprising 87 tasks sourced from real-world clinical data sources across nine languages.
  arXiv  Detail & Related papers  (2025-04-28T04:13:18Z)
• Structured Outputs Enable General-Purpose LLMs to be Medical Experts [50.02627258858336]
  Large language models (LLMs) often struggle with open-ended medical questions. We propose a novel approach utilizing structured medical reasoning. Our approach achieves the highest Factuality Score of 85.8, surpassing fine-tuned models.
  arXiv  Detail & Related papers  (2025-03-05T05:24:55Z)
• Leveraging Large Language Models for Medical Information Extraction and Query Generation [2.1793134762413433]
  This paper introduces a system that integrates large language models (LLMs) into the clinical trial retrieval process. We evaluate six LLMs for query generation, focusing on open-source and relatively small models that require minimal computational resources.
  arXiv  Detail & Related papers  (2024-10-31T12:01:51Z)
• LlamaCare: A Large Medical Language Model for Enhancing Healthcare Knowledge Sharing [0.0]
  We fine-tuned a large language model of medical knowledge with very low carbon emissions and achieved similar performance with ChatGPT by a 24G GPU. We released our processed data for one-shot and few-shot training for some benchmarks such as PubMedQA and USMLE 1-3 step.
  arXiv  Detail & Related papers  (2024-06-04T14:24:53Z)
• OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models [4.556924372105915]
  Open-source (OS) models represent a key area of growth for medical LLMs. We present OpenMedLM, a prompting platform which delivers state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks.
  arXiv  Detail & Related papers  (2024-02-29T17:19:39Z)
• FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability [70.84333325049123]
  FoFo is a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats. This paper presents FoFo, a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats.
  arXiv  Detail & Related papers  (2024-02-28T19:23:27Z)
• Large Language Model Distilling Medication Recommendation Model [58.94186280631342]
  We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs)<n>Our research aims to transform existing medication recommendation methodologies using LLMs.<n>To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
  arXiv  Detail & Related papers  (2024-02-05T08:25:22Z)
• LLM-Rec: Personalized Recommendation via Prompting Large Language Models [62.481065357472964]
  Large language models (LLMs) have showcased their ability to harness commonsense knowledge and reasoning. Recent advances in large language models (LLMs) have showcased their remarkable ability to harness commonsense knowledge and reasoning. This study introduces a novel approach, coined LLM-Rec, which incorporates four distinct prompting strategies of text enrichment for improving personalized text-based recommendations.
  arXiv  Detail & Related papers  (2023-07-24T18:47:38Z)
• An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT [80.33783969507458]
  The 'Impression' section of a radiology report is a critical basis for communication between radiologists and other physicians. Recent studies have achieved promising results in automatic impression generation using large-scale medical text data. These models often require substantial amounts of medical text data and have poor generalization performance.
  arXiv  Detail & Related papers  (2023-04-17T17:13:42Z)
• Large Language Models Encode Clinical Knowledge [21.630872464930587]
  Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. We show that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning.
  arXiv  Detail & Related papers  (2022-12-26T14:28:24Z)
• Information Extraction in Low-Resource Scenarios: Survey and Perspective [56.5556523013924]
  Information Extraction seeks to derive structured information from unstructured texts. This paper presents a review of neural approaches to low-resource IE from emphtraditional and emphLLM-based perspectives.
  arXiv  Detail & Related papers  (2022-02-16T13:44:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.