Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks

• URL: http://arxiv.org/abs/2404.00376v2
• Date: Sun, 30 Jun 2024 09:24:12 GMT
• Title: Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks
• Authors: Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, Jaewoo Kang,
• Abstract summary: We introduce Meerkat, a new family of medical AI systems ranging from 7 to 70 billion parameters. Our systems achieved remarkable accuracy across six medical benchmarks. Meerkat-70B correctly diagnosed 21 out of 38 complex clinical cases, outperforming humans' 13.8.
• Score: 17.40940406100025
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent advancements in commercial large language models (LM) have shown promising results in medical tasks, their closed-source nature poses significant privacy and security concerns, hindering their widespread use in the medical field. Despite efforts to create open-source models, their limited parameters often result in insufficient multi-step reasoning capabilities required for solving complex medical problems. To address this, we introduce Meerkat, a new family of medical AI systems ranging from 7 to 70 billion parameters. The models were trained using our new synthetic dataset consisting of high-quality chain-of-thought reasoning paths sourced from 18 medical textbooks, along with diverse instruction-following datasets. Our systems achieved remarkable accuracy across six medical benchmarks, surpassing the previous best models such as MediTron and BioMistral, and GPT-3.5 by a large margin. Notably, Meerkat-7B surpassed the passing threshold of the United States Medical Licensing Examination (USMLE) for the first time for a 7B-parameter model, while Meerkat-70B outperformed GPT-4 by an average of 1.3%. Additionally, Meerkat-70B correctly diagnosed 21 out of 38 complex clinical cases, outperforming humans' 13.8 and closely matching GPT-4's 21.8. Our systems offered more detailed free-form responses to clinical queries compared to existing small models, approaching the performance level of large commercial models. This significantly narrows the performance gap with large LMs, showcasing its effectiveness in addressing complex medical challenges.

Related papers

• SemiHVision: Enhancing Medical Multimodal Models with a Semi-Human Annotated Dataset and Fine-Tuned Instruction Generation [13.672776832197918]
  Multimodal large language models (MLLMs) have made significant strides, yet they face challenges in the medical domain due to limited specialized knowledge. We seek to address this gap at various stages of the end-to-end learning pipeline, including data collection, model fine-tuning, and evaluation.
  arXiv  Detail & Related papers  (2024-10-19T02:35:35Z)
• Eir: Thai Medical Large Language Models [0.0]
  Eir-8B is a large language model with 8 billion parameters designed to enhance the accuracy of handling medical tasks in the Thai language. Human evaluation was conducted to ensure that the model adheres to care standards and provides unbiased answers. The model is deployed within the hospital's internal network, ensuring both high security and faster processing speeds.
  arXiv  Detail & Related papers  (2024-09-13T04:06:00Z)
• Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data [3.469567586411153]
  Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. This study evaluates the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on a variety of clinical tasks.
  arXiv  Detail & Related papers  (2024-08-25T13:36:22Z)
• Towards Evaluating and Building Versatile Large Language Models for Medicine [57.49547766838095]
  We present MedS-Bench, a benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks.
  arXiv  Detail & Related papers  (2024-08-22T17:01:34Z)
• Capabilities of Gemini Models in Medicine [100.60391771032887]
  We introduce Med-Gemini, a family of highly capable multimodal models specialized in medicine. We evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art (SoTA) performance on 10 of them. Our results offer compelling evidence for Med-Gemini's potential, although further rigorous evaluation will be crucial before real-world deployment.
  arXiv  Detail & Related papers  (2024-04-29T04:11:28Z)
• BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text [82.7001841679981]
  BioMedLM is a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with larger models. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics.
  arXiv  Detail & Related papers  (2024-03-27T10:18:21Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models [91.25119823784705]
  Large language models (LLMs) can potentially democratize access to medical knowledge. We release MEDITRON: a suite of open-source LLMs with 7B and 70B parameters adapted to the medical domain.
  arXiv  Detail & Related papers  (2023-11-27T18:49:43Z)
• AlpaCare:Instruction-tuned Large Language Models for Medical Application [23.697610908951443]
  We propose creating a diverse, machine-generated medical IFT dataset, MedInstruct-52k, using GPT-4 and ChatGPT. We then fine-tune LLaMA-series models on the dataset to develop AlpaCare. Despite using a smaller domain-specific dataset, AlpaCare demonstrates superior performance on medical applications.
  arXiv  Detail & Related papers  (2023-10-23T04:22:50Z)
• Federated Learning Enables Big Data for Rare Cancer Boundary Detection [98.5549882883963]
  We present findings from the largest Federated ML study to-date, involving data from 71 healthcare institutions across 6 continents. We generate an automatic tumor boundary detector for the rare disease of glioblastoma. We demonstrate a 33% improvement over a publicly trained model to delineate the surgically targetable tumor, and 23% improvement over the tumor's entire extent.
  arXiv  Detail & Related papers  (2022-04-22T17:27:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.