Towards Building Multilingual Language Model for Medicine

• URL: http://arxiv.org/abs/2402.13963v4
• Date: Sun, 2 Jun 2024 10:02:00 GMT
• Title: Towards Building Multilingual Language Model for Medicine
• Authors: Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, Weidi Xie,
• Abstract summary: We construct a multilingual medical corpus, containing approximately 25.5B tokens encompassing 6 main languages. We propose a multilingual medical multi-choice question-answering benchmark with rationale, termed as MMedBench. Our final model, MMed-Llama 3, with only 8B parameters, achieves superior performance compared to all other open-source models on both MMedBench and English benchmarks.
• Score: 54.1382395897071
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The development of open-source, multilingual medical language models can benefit a wide, linguistically diverse audience from different regions. To promote this domain, we present contributions from the following: First, we construct a multilingual medical corpus, containing approximately 25.5B tokens encompassing 6 main languages, termed as MMedC, enabling auto-regressive domain adaptation for general LLMs; Second, to monitor the development of multilingual medical LLMs, we propose a multilingual medical multi-choice question-answering benchmark with rationale, termed as MMedBench; Third, we have assessed a number of open-source large language models (LLMs) on our benchmark, along with those further auto-regressive trained on MMedC. Our final model, MMed-Llama 3, with only 8B parameters, achieves superior performance compared to all other open-source models on both MMedBench and English benchmarks, even rivaling GPT-4. In conclusion, in this work, we present a large-scale corpus, a benchmark and a series of models to support the development of multilingual medical LLMs.

Related papers

• EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models [50.459861376459656]
  EMMA-500 is a large-scale multilingual language model continue-trained on texts across 546 languages. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity.
  arXiv  Detail & Related papers  (2024-09-26T14:40:45Z)
• Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach [6.921012069327385]
  Open-source, multilingual medical large language models (LLMs) have the potential to serve linguistically diverse populations across different regions. We introduce two multilingual instruction fine-tuning datasets, containing over 200k high-quality medical samples in six languages. We propose a two-stage training paradigm: the first stage injects general medical knowledge using MMed-IFT, while the second stage fine-tunes task-specific multiple-choice questions with MMed-IFT-MC.
  arXiv  Detail & Related papers  (2024-09-09T15:42:19Z)
• Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain [19.58987478434808]
  We present Medical mT5, the first open-source text-to-text multilingual model for the medical domain. A comprehensive evaluation shows that Medical mT5 outperforms both encoders and similarly sized text-to-text models for the Spanish, French, and Italian benchmarks.
  arXiv  Detail & Related papers  (2024-04-11T10:01:32Z)
• Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People [68.59917533894608]
  We aim to develop medical LLMs across the six most widely spoken languages, encompassing a global population of 6.1 billion. This effort culminates in the creation of the ApolloCorpora multilingual medical dataset and the XMedBench benchmark. We will open-source training corpora, code, model weights and evaluation benchmark.
  arXiv  Detail & Related papers  (2024-03-06T11:56:02Z)
• OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models [59.54423478596468]
  We introduce OMGEval, the first Open-source Multilingual Generative test set that can assess the capability of LLMs in different languages. For each language, OMGEval provides 804 open-ended questions, covering a wide range of important capabilities of LLMs. Specifically, the current version of OMGEval includes 5 languages (i.e., Zh, Ru, Fr, Es, Ar)
  arXiv  Detail & Related papers  (2024-02-21T04:42:41Z)
• BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains [8.448541067852]
  Large Language Models (LLMs) have demonstrated remarkable versatility in recent years. Despite the availability of various open-source LLMs tailored for health contexts, adapting general-purpose LLMs to the medical domain presents significant challenges. We introduce BioMistral, an open-source LLM tailored for the biomedical domain, utilizing Mistral as its foundation model.
  arXiv  Detail & Related papers  (2024-02-15T23:39:04Z)
• KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model [37.69464822182714]
  Most biomedical pretrained language models are monolingual and cannot handle the growing cross-lingual requirements. We propose a model called KBioXLM, which transforms the multilingual pretrained model XLM-R into the biomedical domain using a knowledge-anchored approach.
  arXiv  Detail & Related papers  (2023-11-20T07:02:35Z)
• PolyLM: An Open Source Polyglot Large Language Model [57.64420154135178]
  We present PolyLM, a multilingual large language model (LLMs) trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-12T09:00:37Z)
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning [77.33262578776291]
  Previous work has demonstrated that machine translation systems can be created by finetuning on bitext. We show that multilingual translation models can be created through multilingual finetuning. We demonstrate that pretrained models can be extended to incorporate additional languages without loss of performance.
  arXiv  Detail & Related papers  (2020-08-02T05:36:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.