MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval

• URL: http://arxiv.org/abs/2606.24200v1
• Date: Tue, 23 Jun 2026 06:41:13 GMT
• Title: MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval
• Abstract summary: multilingual medical retrieval demands three capabilities: cross-lingual alignment, concept discrimination, and evidence retrieval.<n>We introduce MMed-Bench-IR, a benchmark designed to disentangle these axes across 6 languages and three structurally heterogeneous tasks.<n>The three tasks share zero concept and query overlap by design, ensuring that aggregate scores reflect genuine capability breadth.
• Score: 2.0221473384338884
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-augmented generation (RAG) in clinical settings increasingly requires multilingual retrieval against predominantly English evidence corpora. Multilingual medical retrieval demands three capabilities: cross-lingual alignment, concept discrimination, and evidence retrieval. However, existing benchmarks evaluate these only in isolation, leaving the interaction between biomedical expertise and multilingual coverage unmeasured. We introduce MMed-Bench-IR, a benchmark designed to disentangle these axes across 6 languages and three structurally heterogeneous tasks: (1) cross-lingual medical QA retrieval with 6,127 queries grounded in the Unified Medical Language System (UMLS), (2) concept discrimination over 4,975 confusion sets at three difficulty tiers, and (3) multilingual evidence retrieval for RAG with 2,040 quality-assured queries. The three tasks share zero concept and query overlap by design, ensuring that aggregate scores reflect genuine capability breadth. Evaluation of ten systems across six paradigm families reveals severe cross-lingual failure: biomedical encoders that score 0.818 nDCG@10 in English drop to 0.056 in Japanese, a gap that English-only benchmarks cannot detect.

Related papers

• HealMed: Multilingual Evaluation of Large Language Models in Medicine [82.79641241632788]
  We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine.<n>The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions.<n>The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps.
  arXiv  Detail & Related papers  (2026-08-20T12:52:09Z)
• OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents [32.227608559573525]
  We introduce OmnilingualGAIA2, a machine-translated expansion of the GAIA2 agentic benchmark, covering ten target languages spanning five writing systems.<n>We find a universal cross-lingual gap of 8.8-18.4 pass@3 points that is agent-asymmetric in magnitude, concentrates on tool-orchestration rather than quantitative reasoning, and does not close with model scale.<n>Human-expert linguistic analysis further identifies morphological cue loss and amplified ambiguity as the primary failure mechanisms in non-Latin-script languages.
  arXiv  Detail & Related papers  (2026-08-09T15:49:00Z)
• The Multilingual Curse at the Retrieval Layer: Evidence from Amharic [49.25758237235551]
  Strong zero-shot scores on multilingual benchmarks are often taken as evidence that current encoders transfer reliably across many languages.<n>We argue that this assumption breaks down for underrepresented, morphologically rich languages.<n>We compare zero-shot multilingual retrievers, Amharic-fine-tuned multilingual retrievers, and monolingual Amharic retrievers.
  arXiv  Detail & Related papers  (2026-05-23T12:44:30Z)
• JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models [47.20100799532625]
  We introduce JMedEthicBench, the first multi-turn conversational benchmark for evaluating medical safety of Large Language Models.<n>Using a dual-LLM scoring protocol, we evaluate 27 models and find that commercial models maintain robust safety while medical-specialized models exhibit increased vulnerability.
  arXiv  Detail & Related papers  (2026-01-04T18:18:18Z)
• MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts [0.0]
  Large language models (LLMs) show increasing promise in medical applications, but their ability to detect and correct errors in clinical texts remains under-evaluated.<n>We introduce MedRECT, a cross-lingual benchmark (Japanese/English) that formulates medical error handling as three subtasks.<n>We evaluate 9 contemporary LLMs spanning proprietary, open-weight, and reasoning families.
  arXiv  Detail & Related papers  (2025-11-01T06:19:34Z)
• Filling in the Clinical Gaps in Benchmark: Case for HealthBench for the Japanese medical system [5.7880565661958565]
  This study investigates the applicability of HealthBench to the Japanese context.<n> resources in Japanese are scarce and often consist of translated multiple-choice questions.
  arXiv  Detail & Related papers  (2025-09-22T07:36:12Z)
• Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs [3.1894617416005855]
  Large language models (LLMs) present a promising solution to automate various ophthalmology procedures.<n>LLMs have demonstrated significantly varied performance across different languages in natural language question-answering tasks.<n>This study introduces the first multilingual ophthalmological question-answering benchmark with manually curated questions parallel across languages.
  arXiv  Detail & Related papers  (2024-12-18T20:18:03Z)
• Towards Building Multilingual Language Model for Medicine [54.1382395897071]
  We construct a multilingual medical corpus, containing approximately 25.5B tokens encompassing 6 main languages. We propose a multilingual medical multi-choice question-answering benchmark with rationale, termed as MMedBench. Our final model, MMed-Llama 3, with only 8B parameters, achieves superior performance compared to all other open-source models on both MMedBench and English benchmarks.
  arXiv  Detail & Related papers  (2024-02-21T17:47:20Z)
• A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models [57.88111980149541]
  We introduce Asclepius, a novel Med-MLLM benchmark that assesses Med-MLLMs in terms of distinct medical specialties and different diagnostic capacities.<n>Grounded in 3 proposed core principles, Asclepius ensures a comprehensive evaluation by encompassing 15 medical specialties.<n>We also provide an in-depth analysis of 6 Med-MLLMs and compare them with 3 human specialists.
  arXiv  Detail & Related papers  (2024-02-17T08:04:23Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.