HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs

• URL: http://arxiv.org/abs/2311.09774v2
• Date: Sun, 15 Sep 2024 08:41:01 GMT
• Title: HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs
• Authors: Junying Chen, Xidong Wang, Ke Ji, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xiang Wan, Haizhou Li, Benyou Wang,
• Abstract summary: HuatuoGPT-II has shown state-of-the-art performance in Chinese medicine domain on a number of benchmarks. It even outperforms proprietary models like ChatGPT and GPT-4 in some aspects, especially in Traditional Chinese Medicine.
• Score: 61.41790586411816
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adapting a language model into a specific domain, a.k.a `domain adaption', is a common practice when specialized knowledge, e.g. medicine, is not encapsulated in a general language model like Llama2. The challenge lies in the heterogeneity of data across the two training stages, as it varies in languages, genres, or formats. To tackle this and simplify the learning protocol, we propose to transform heterogeneous data, from the both pre-training and supervised stages, into a unified, simple input-output pair format. We validate the new protocol in the domains where proprietary LLMs like ChatGPT perform relatively poorly, such as Traditional Chinese Medicine. The developed model, HuatuoGPT-II, has shown state-of-the-art performance in Chinese medicine domain on a number of benchmarks, e.g. medical licensing exams. It even outperforms proprietary models like ChatGPT and GPT-4 in some aspects, especially in Traditional Chinese Medicine. Expert manual evaluations further validate HuatuoGPT-II's advantages over existing LLMs. Notably, HuatuoGPT-II was benchmarked in a fresh Chinese National Medical Licensing Examination where it achieved the best performance, showcasing not only its effectiveness but also its generalization capabilities.

Related papers

• Large Language Model Distilling Medication Recommendation Model [61.89754499292561]
  We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs) Our research aims to transform existing medication recommendation methodologies using LLMs. To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
  arXiv  Detail & Related papers  (2024-02-05T08:25:22Z)
• ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences [51.66185471742271]
  We propose ChiMed-GPT, a benchmark LLM designed explicitly for Chinese medical domain. ChiMed-GPT undergoes a comprehensive training regime with pre-training, SFT, and RLHF. We analyze possible biases through prompting ChiMed-GPT to perform attitude scales regarding discrimination of patients.
  arXiv  Detail & Related papers  (2023-11-10T12:25:32Z)
• Continuous Training and Fine-tuning for Domain-Specific Language Models in Medical Question Answering [4.254954312483959]
  Large language models exhibit promising general capabilities but often lack specialized knowledge for domain-specific tasks. This work demonstrates a method using continuous training and instruction fine-tuning to rapidly adapt Llama 2 base models to the Chinese medical domain.
  arXiv  Detail & Related papers  (2023-11-01T00:18:00Z)
• Efficient Finetuning Large Language Models For Vietnamese Chatbot [1.2075778142867704]
  Large language models (LLMs) have been shown to achieve remarkable performance across a variety of natural language tasks. We leverage large-scale instruction-following datasets from open-source projects, namely Alpaca, GPT4All, and Chat-Doctor. We utilize parameter-efficient tuning through Low-Rank Adaptation (LoRA) on two open LLMs, resulting four models: Bloomz-Chat, Bloomz-Doctor, GPTJ-Chat, GPTJ-Doctor.
  arXiv  Detail & Related papers  (2023-09-09T00:11:53Z)
• Large Language Models Streamline Automated Machine Learning for Clinical Studies [2.4889420816783963]
  ChatGPT Advanced Data Analysis (ADA) is an extension of GPT-4 to perform machine learning analyses efficiently. ADA autonomously developed state-of-the-art ML models based on the original study's training data to predict clinical outcomes. Strikingly, the ChatGPT ADA-crafted ML models often outperformed their counterparts.
  arXiv  Detail & Related papers  (2023-08-27T14:28:38Z)
• Customizing General-Purpose Foundation Models for Medical Report Generation [64.31265734687182]
  The scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks. We propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs) in computer vision and natural language processing.
  arXiv  Detail & Related papers  (2023-06-09T03:02:36Z)
• DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains [4.989459243399296]
  We propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data.
  arXiv  Detail & Related papers  (2023-04-03T13:25:53Z)
• Detecting Text Formality: A Study of Text Classification Approaches [78.11745751651708]
  This work proposes the first to our knowledge systematic study of formality detection methods based on statistical, neural-based, and Transformer-based machine learning methods. We conducted three types of experiments -- monolingual, multilingual, and cross-lingual. The study shows the overcome of Char BiLSTM model over Transformer-based ones for the monolingual and multilingual formality classification task.
  arXiv  Detail & Related papers  (2022-04-19T16:23:07Z)
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing [73.37262264915739]
  We show that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks.
  arXiv  Detail & Related papers  (2020-07-31T00:04:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.