HealMed: Multilingual Evaluation of Large Language Models in Medicine

• URL: http://arxiv.org/abs/2608.19981v1
• Date: Thu, 20 Aug 2026 12:52:09 GMT
• Title: HealMed: Multilingual Evaluation of Large Language Models in Medicine
• Abstract summary: We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine.<n>The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions.<n>The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps.
• Score: 82.79641241632788
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats: MCQA, NLI and open-ended QA. The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions. Each translation was evaluated and revised by two experts fluent in English and the corresponding target language. On HealMed, performance declined most in low-resource languages, although the size of the gap varied markedly across languages and models. The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps. Medical specialization alone did not ensure multilingual robustness. Furthermore, expert revision could either raise or lower measured performance, indicating that translation quality materially affects cross-language evaluation results.

Related papers

• Effects of Cross-lingual Evidence in Multilingual Medical Question Answering [14.325363876249972]
  This paper investigates Multilingual Medical Question Answering across high-resource (English, Spanish, French, Italian) and low-resource (Basque, Kazakh) languages.<n>We evaluate three types of external evidence sources across models of varying size.
  arXiv  Detail & Related papers  (2026-04-22T13:09:41Z)
• MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark [8.428847258506176]
  Arabic is one of the most underrepresented languages in natural language processing research.<n>MedAraBench is a large-scale dataset consisting of Arabic multiple-choice question-answer pairs across various medical specialties.
  arXiv  Detail & Related papers  (2026-02-02T06:52:20Z)
• Toward Global Large Language Models in Medicine [67.38063166560406]
  GlobMed is a large multilingual medical dataset containing over 500,000 entries spanning 12 languages, including four low-resource languages.<n>GlobMed-Bench assesses 56 state-of-the-art proprietary and open-weight LLMs across multiple multilingual medical tasks, revealing significant performance disparities across languages.<n>GlobMed-LLMs achieved an average performance improvement of over 40% relative to baseline models, with a more than threefold increase in performance on low-resource languages.
  arXiv  Detail & Related papers  (2026-01-05T15:05:49Z)
• PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark [3.2640411992544345]
  Large Language Models (LLMs) have achieved remarkable performance on a wide range of Natural Language Processing (NLP) benchmarks.<n>We introduce PersianMedQA, a large-scale dataset of 20,785 expert-validated multiple-choice Persian medical questions from 14 years of Iranian national medical exams.<n>We benchmark 40 state-of-the-art models, including general-purpose, Persian fine-tuned, and medical LLMs, in zero-shot and chain-of-thought settings.
  arXiv  Detail & Related papers  (2025-05-30T21:34:30Z)
• MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation [12.644932351174786]
  We release MultiMed-ST, a large-scale ST dataset for the medical domain, spanning all translation directions in five languages.<n>With 290,000 samples, this is the largest medical MT dataset and the largest many-to-many multilingual ST among all domains.<n>We present the most comprehensive ST analysis in the field's history, to our best knowledge, including: empirical baselines, bilingual-multilingual comparative study, end-to-end vs. cascaded comparative study, task-specific vs. multi-task sequence-to-sequence comparative study, code-switch analysis, and quantitative-qualitative
  arXiv  Detail & Related papers  (2025-04-04T15:49:17Z)
• BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities [46.684778713683976]
  BiMediX2 is a bilingual (Arabic-English) Bio-Medical EXpert Large Multimodal Model.<n>It supports text-based and image-based medical interactions.<n>It enables multi-turn conversation in Arabic and English.
  arXiv  Detail & Related papers  (2024-12-10T18:59:35Z)
• Towards Building Multilingual Language Model for Medicine [54.1382395897071]
  We construct a multilingual medical corpus, containing approximately 25.5B tokens encompassing 6 main languages. We propose a multilingual medical multi-choice question-answering benchmark with rationale, termed as MMedBench. Our final model, MMed-Llama 3, with only 8B parameters, achieves superior performance compared to all other open-source models on both MMedBench and English benchmarks.
  arXiv  Detail & Related papers  (2024-02-21T17:47:20Z)
• BiMediX: Bilingual Medical Mixture of Experts LLM [90.3257333861513]
  We introduce BiMediX, the first bilingual medical mixture of experts LLM designed for seamless interaction in both English and Arabic.<n>Our model facilitates a wide range of medical interactions in English and Arabic, including multi-turn chats to inquire about additional details.<n>We propose a semi-automated English-to-Arabic translation pipeline with human refinement to ensure high-quality translations.
  arXiv  Detail & Related papers  (2024-02-20T18:59:26Z)
• MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation [22.986061896641083]
  MedEval is a multi-level, multi-task, and multi-domain medical benchmark to facilitate the development of language models for healthcare. With 22,779 collected sentences and 21,228 reports, we provide expert annotations at multiple levels, offering a granular potential usage of the data.
  arXiv  Detail & Related papers  (2023-10-21T18:59:41Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.