JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation

• URL: http://arxiv.org/abs/2605.22080v2
• Date: Thu, 28 May 2026 03:03:05 GMT
• Title: JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation
• Abstract summary: We introduce JMed48k, a Japanese healthcare licensing benchmark for evaluating vision-language models.<n>Built from official PDF materials released by the Japanese Ministry of Health, Labour and Welfare, JMed48k contains 48,862 exam questions and 20,142 images.<n>We derive JMed48k-Eval, a recent five-year evaluation subset with 12,484 scored questions, including 9,905 text-only questions and 2,579 questions with images.
• Score: 53.571125769505194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce JMed48k, a multi-profession Japanese healthcare licensing benchmark for evaluating vision-language models. Built from official PDF materials released by the Japanese Ministry of Health, Labour and Welfare, JMed48k contains 48,862 exam questions and 20,142 images from 11 national licensing examinations between 2005 and 2025, with visual content annotated under an 8-type taxonomy. From this corpus, we derive JMed48k-Eval, a recent five-year evaluation subset with 12,484 scored questions, including 9,905 text-only questions and 2,579 questions with images. We evaluate 21 proprietary, open-source, and medical-specific models, reporting text-only and with-image performance separately. Because these subsets contain different questions, we further introduce a paired image-removal audit that evaluates questions with images before and after removing visual content to explore four answer-transition states. The audit shows that proprietary and open source models gain substantially from images, whereas medical-specific systems show limited observable use of visual evidence, with many correct answers persisting after image removal. Even among proprietary models, the net image-removal effect varies sevenfold across professions, from +5.7 points on Physician questions to +39.8 points on Public Health Nurse questions. We release JMed48k to support reproducible, profession-stratified evaluation of vision-language models in medical licensing settings.

Related papers

• Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence [2.779050345468656]
  The benchmark comprises image-containing questions spanning diverse medical specialties and visual domains.<n>We evaluate Polish-oriented, general-purpose open-weight, and commercial vision-language models.
  arXiv  Detail & Related papers  (2026-08-13T08:07:22Z)
• A Vision-language Framework for Comparative Reasoning in Radiology [68.52471827773482]
  We formulate radiological comparison as an entity-aware cross-image reasoning problem.<n>We construct a large-scale comparative imaging resource derived from routine image-report pairs.<n>We develop MedReCo, an entity-aware visual encoder for controllable retrieval of clinically analogous cases.
  arXiv  Detail & Related papers  (2026-06-04T17:12:47Z)
• ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding [33.868464991344354]
  We introduce ReXInTheWild, a benchmark of 955 clinician-verified multiple-choice questions spanning seven clinical topics.<n> Gemini-3 achieves 78% accuracy, followed by Claude Opus 4.5 (72%) and GPT-5 (68%), while the medical specialist model MedGemma reaches only 37%.<n>A systematic error analysis also reveals four categories of common errors, ranging from low-level geometric errors to high-level reasoning failures.
  arXiv  Detail & Related papers  (2026-03-19T22:54:28Z)
• KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination [16.50828571559655]
  KorMedMCQA-V is a Korean medical licensing-exam-style multimodal multiple-choice question answering benchmark.<n>The dataset consists of 1,534 questions with 2,043 associated images from Korean Medical Licensing Examinations.
  arXiv  Detail & Related papers  (2026-02-14T07:42:04Z)
• Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering [27.308408027453012]
  We investigate whether large vision language models (VLMs) demonstrate genuine visual grounding when answering Italian medical questions.<n>We test four state-of-the-art models: Claude Sonnet 4.5, GPT-4o, GPT-5-mini, and Gemini 2.0 flash exp.
  arXiv  Detail & Related papers  (2025-11-24T15:26:58Z)
• Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding [112.46150793476603]
  We introduce Hulu-Med, a transparent, generalist medical Vision-Language Model (VLM)<n>Hulu-Med is trained on a curated corpus of 16.7 million samples, spanning 12 major anatomical systems and 14 medical imaging modalities.<n>Hulu-Med surpasses existing open-source models on 27 of 30 benchmarks and outperforms proprietary systems such as GPT-4o on 16 benchmarks.
  arXiv  Detail & Related papers  (2025-10-09T17:06:42Z)
• TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models [54.48710348910535]
  Existing medical reasoning benchmarks primarily focus on analyzing a patient's condition based on an image from a single visit.<n>We introduce TemMed-Bench, the first benchmark designed for analyzing changes in patients' conditions between different clinical visits.
  arXiv  Detail & Related papers  (2025-09-29T17:51:26Z)
• KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations [6.453078564406654]
  KokushiMD-10 is the first multimodal benchmark constructed from ten Japanese national healthcare licensing exams.<n>This benchmark spans multiple fields, including Medicine, Dentistry, Nursing, Pharmacy, and allied health professions.<n>It contains over 11588 real exam questions, incorporating clinical images and expert-annotated rationales to evaluate both textual and visual reasoning.
  arXiv  Detail & Related papers  (2025-06-09T02:26:02Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• Towards the Use of Saliency Maps for Explaining Low-Quality Electrocardiograms to End Users [51.644376281196394]
  When using medical images for diagnosis, it is important that the images are of high quality.<n>In telemedicine, a common problem is that the quality issue is only flagged once the patient has left the clinic, meaning they must return in order to have the exam redone.<n>This paper reports on the development of an AI system for flagging and explaining low-quality medical images in real-time.
  arXiv  Detail & Related papers  (2022-07-06T14:53:26Z)
• A Multilingual Neural Machine Translation Model for Biomedical Data [84.17747489525794]
  We release a multilingual neural machine translation model, which can be used to translate text in the biomedical domain. The model can translate from 5 languages (French, German, Italian, Korean and Spanish) into English. It is trained with large amounts of generic and biomedical data, using domain tags.
  arXiv  Detail & Related papers  (2020-08-06T21:26:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.