Towards Training A Chinese Large Language Model for Anesthesiology
- URL: http://arxiv.org/abs/2403.02742v1
- Date: Tue, 5 Mar 2024 07:53:49 GMT
- Title: Towards Training A Chinese Large Language Model for Anesthesiology
- Authors: Zhonghai Wang, Jie Jiang, Yibing Zhan, Bohao Zhou, Yanhong Li, Chong
Zhang, Liang Ding, Hua Jin, Jun Peng, Xu Lin, and Weifeng Liu
- Abstract summary: We introduce a Chinese Anesthesia model built upon existing medical large language models, e.g., Llama.
Hypnos' contributions have three aspects: 1) The data, such as utilizing Self-Instruct, acquired from current LLMs likely includes inaccuracies.
Hypnos employs a general-to-specific training strategy that starts by fine-tuning LLMs using the general medicine data and subsequently improving the fine-tuned LLMs using data specifically from Anesthesiology.
- Score: 37.44529879903248
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Medical large language models (LLMs) have gained popularity recently due to
their significant practical utility. However, most existing research focuses on
general medicine, and there is a need for in-depth study of LLMs in specific
fields like anesthesiology. To fill the gap, we introduce Hypnos, a Chinese
Anesthesia model built upon existing LLMs, e.g., Llama. Hypnos' contributions
have three aspects: 1) The data, such as utilizing Self-Instruct, acquired from
current LLMs likely includes inaccuracies. Hypnos implements a cross-filtering
strategy to improve the data quality. This strategy involves using one LLM to
assess the quality of the generated data from another LLM and filtering out the
data with low quality. 2) Hypnos employs a general-to-specific training
strategy that starts by fine-tuning LLMs using the general medicine data and
subsequently improving the fine-tuned LLMs using data specifically from
Anesthesiology. The general medical data supplement the medical expertise in
Anesthesiology and enhance the effectiveness of Hypnos' generation. 3) We
introduce a standardized benchmark for evaluating medical LLM in
Anesthesiology. Our benchmark includes both publicly available instances from
the Internet and privately obtained cases from the Hospital. Hypnos outperforms
other medical LLMs in anesthesiology in metrics, GPT-4, and human evaluation on
the benchmark dataset.
Related papers
- Developing Healthcare Language Model Embedding Spaces [0.20971479389679337]
Pre-trained Large Language Models (LLMs) often struggle on out-of-domain datasets like healthcare focused text.
Three methods are assessed: traditional masked language modeling, Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR) and a novel pre-training objective utilizing metadata categories from the healthcare settings.
Contrastively trained models outperform other approaches on the classification tasks, delivering strong performance from limited labeled data and with fewer model parameter updates required.
arXiv Detail & Related papers (2024-03-28T19:31:32Z) - OpenMedLM: Prompt engineering can out-perform fine-tuning in medical
question-answering with open-source large language models [4.556924372105915]
Open-source (OS) models represent a key area of growth for medical LLMs.
We present OpenMedLM, a prompting platform which delivers state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks.
arXiv Detail & Related papers (2024-02-29T17:19:39Z) - Large Language Model Distilling Medication Recommendation Model [61.89754499292561]
We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs)
Our research aims to transform existing medication recommendation methodologies using LLMs.
To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
arXiv Detail & Related papers (2024-02-05T08:25:22Z) - From Beginner to Expert: Modeling Medical Knowledge into General LLMs [22.475129648458136]
Large language model (LLM) based artificial intelligence (AI) systems have demonstrated remarkable capabilities in natural language understanding and generation.
These models face a significant challenge when it comes to sensitive applications, such as reasoning over medical knowledge and answering medical questions in a physician-like manner.
In this work, we start from a pre-trained general LLM model (AntGLM-10B) and fine-tune it from a medical beginner towards a medical expert (called AntGLM-Med-10B)
arXiv Detail & Related papers (2023-12-02T05:54:06Z) - MEDITRON-70B: Scaling Medical Pretraining for Large Language Models [91.25119823784705]
Large language models (LLMs) can potentially democratize access to medical knowledge.
We release MEDITRON: a suite of open-source LLMs with 7B and 70B parameters adapted to the medical domain.
arXiv Detail & Related papers (2023-11-27T18:49:43Z) - ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences [51.66185471742271]
We propose ChiMed-GPT, a benchmark LLM designed explicitly for Chinese medical domain.
ChiMed-GPT undergoes a comprehensive training regime with pre-training, SFT, and RLHF.
We analyze possible biases through prompting ChiMed-GPT to perform attitude scales regarding discrimination of patients.
arXiv Detail & Related papers (2023-11-10T12:25:32Z) - A Survey of Large Language Models in Medicine: Progress, Application, and Challenge [85.09998659355038]
Large language models (LLMs) have received substantial attention due to their capabilities for understanding and generating human language.
This review aims to provide a detailed overview of the development and deployment of LLMs in medicine.
arXiv Detail & Related papers (2023-11-09T02:55:58Z) - Augmenting Black-box LLMs with Medical Textbooks for Clinical Question
Answering [54.13933019557655]
We present a system called LLMs Augmented with Medical Textbooks (LLM-AMT)
LLM-AMT integrates authoritative medical textbooks into the LLMs' framework using plug-and-play modules.
We found that medical textbooks as a retrieval corpus is proven to be a more effective knowledge database than Wikipedia in the medical domain.
arXiv Detail & Related papers (2023-09-05T13:39:38Z) - Enhancing Small Medical Learners with Privacy-preserving Contextual Prompting [24.201549275369487]
We present a method that harnesses large language models' medical expertise to boost SLM performance in medical tasks under privacy-restricted scenarios.
Specifically, we mitigate patient privacy issues by extracting keywords from medical data and prompting the LLM to generate a medical knowledge-intensive context.
Our method significantly enhances performance in both few-shot and full training settings across three medical knowledge-intensive tasks.
arXiv Detail & Related papers (2023-05-22T05:14:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.