CareBot: A Pioneering Full-Process Open-Source Medical Language Model

• URL: http://arxiv.org/abs/2412.15236v2
• Date: Mon, 23 Dec 2024 02:44:18 GMT
• Title: CareBot: A Pioneering Full-Process Open-Source Medical Language Model
• Authors: Lulu Zhao, Weihao Zeng, Xiaofeng Shi, Hua Zhou,
• Abstract summary: CareBot is a bilingual medical LLM that integrates continuous pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning with human feedback (RLHF) DataRater is a model designed to assess data quality during CPT, ensuring that the training data is both accurate and relevant. Our rigorous evaluations on Chinese and English benchmarks confirm CareBot's effectiveness in medical consultation and education.
• Score: 8.868481107848185
• License:
• Abstract: Recently, both closed-source LLMs and open-source communities have made significant strides, outperforming humans in various general domains. However, their performance in specific professional domains such as medicine, especially within the open-source community, remains suboptimal due to the complexity of medical knowledge. In this paper, we propose CareBot, a bilingual medical LLM, which leverages a comprehensive approach integrating continuous pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning with human feedback (RLHF). Our novel two-stage CPT method, comprising Stable CPT and Boost CPT, effectively bridges the gap between general and domain-specific data, facilitating a smooth transition from pre-training to fine-tuning and enhancing domain knowledge progressively. We also introduce DataRater, a model designed to assess data quality during CPT, ensuring that the training data is both accurate and relevant. For SFT, we develope a large and diverse bilingual dataset, along with ConFilter, a metric to enhance multi-turn dialogue quality, which is crucial to improving the model's ability to handle more complex dialogues. The combination of high-quality data sources and innovative techniques significantly improves CareBot's performance across a range of medical applications. Our rigorous evaluations on Chinese and English benchmarks confirm CareBot's effectiveness in medical consultation and education. These advancements not only address current limitations in medical LLMs but also set a new standard for developing effective and reliable open-source models in the medical domain. We will open-source the datasets and models later, contributing valuable resources to the research community.

Related papers

• Continually Evolved Multimodal Foundation Models for Cancer Prognosis [50.43145292874533]
  Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates. Previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information. Existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals. Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities.
  arXiv  Detail & Related papers  (2025-01-30T06:49:57Z)
• STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering [58.79671189792399]
  STLLaVA-Med is designed to train a policy model capable of auto-generating medical visual instruction data. We validate the efficacy and data efficiency of STLLaVA-Med across three major medical Visual Question Answering (VQA) benchmarks.
  arXiv  Detail & Related papers  (2024-06-28T15:01:23Z)
• Aqulia-Med LLM: Pioneering Full-Process Open-Source Medical Language Models [8.252044870864523]
  We propose Aquila-Med, a bilingual medical LLM based on Aquila. We construct a large-scale Chinese and English medical dataset for continue pre-training and a high-quality SFT dataset. Aquila-Med achieves notable results across single-turn, multi-turn dialogues, and medical multiple-choice questions.
  arXiv  Detail & Related papers  (2024-06-18T01:30:07Z)
• LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery [57.358568111574314]
  Patient data privacy often restricts the availability of old data when updating the model. Prior CL studies overlooked two vital problems in the surgical domain. This paper proposes addressing these problems with a multimodal large language model (LLM) and an adaptive weight assignment methodology.
  arXiv  Detail & Related papers  (2024-02-26T15:35:24Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• Natural Language Programming in Medicine: Administering Evidence Based Clinical Workflows with Autonomous Agents Powered by Generative Large Language Models [29.05425041393475]
  Generative Large Language Models (LLMs) hold significant promise in healthcare. This study assessed the potential of LLMs to function as autonomous agents in a simulated tertiary care medical center.
  arXiv  Detail & Related papers  (2024-01-05T15:09:57Z)
• README: Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLP [9.432205523734707]
  We introduce a new task of automatically generating lay definitions, aiming to simplify medical terms into patient-friendly lay language. We first created the dataset, an extensive collection of over 50,000 unique (medical term, lay definition) pairs and 300,000 mentions. We have also engineered a data-centric Human-AI pipeline that synergizes data filtering, augmentation, and selection to improve data quality.
  arXiv  Detail & Related papers  (2023-12-24T23:01:00Z)
• Multi-Site Clinical Federated Learning using Recursive and Attentive Models and NVFlare [13.176351544342735]
  This paper develops an integrated framework that addresses data privacy and regulatory compliance challenges. It includes the development of an integrated framework that addresses data privacy and regulatory compliance challenges while maintaining elevated accuracy and substantiating the efficacy of the proposed approach.
  arXiv  Detail & Related papers  (2023-06-28T17:00:32Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources [40.32772510980854]
  "patchwork learning" (PL) is a paradigm that integrates information from disparate datasets composed of different data modalities. PL allows the simultaneous utilization of complementary data sources while preserving data privacy. We present the concept of patchwork learning and its current implementations in healthcare, exploring the potential opportunities and applicable data sources.
  arXiv  Detail & Related papers  (2023-05-10T14:50:33Z)
• Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matching [49.78442796596806]
  We propose an innovative privacy-aware data augmentation approach for patient-trial matching (LLM-PTM) Our experiments demonstrate a 7.32% average improvement in performance using the proposed LLM-PTM method, and the generalizability to new data is improved by 12.12%.
  arXiv  Detail & Related papers  (2023-03-24T03:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.