Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study

• URL: http://arxiv.org/abs/2606.03693v1
• Date: Tue, 02 Jun 2026 14:14:27 GMT
• Title: Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study
• Abstract summary: We introduce IndoRad-VQA, an Indonesian adaptation of VQA-RAD, to assess whether medical VLMs retain radiology reasoning ability when questions are asked in Bahasa Indonesia.<n>We evaluate general-purpose, Southeast Asian multilingual, and medical-specific VLMs under English and prompting settings.<n>Our findings show that strong performance on English medical VQA benchmarks does not necessarily translate to robust behavior in Indonesian clinical contexts.
• Score: 2.8292841621378844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical Vision-Language Models (VLMs) are typically evaluated on English radiology visual question answering benchmarks, leaving their robustness under non-English clinical language largely unexplored. We introduce IndoRad-VQA, an Indonesian adaptation of VQA-RAD, to assess whether medical VLMs retain radiology reasoning ability when questions are asked in Bahasa Indonesia. Radiology question-answer pairs are translated into Indonesian with self-evaluation-based quality control to preserve clinical meaning, terminology consistency, and answer equivalence. We evaluate general-purpose, Southeast Asian multilingual, and medical-specific VLMs under English and Indonesian prompting settings. Beyond accuracy, we quantify the language robustness gap between English and Indonesian inputs. We also conduct an error analysis to identify failure modes of question answering, such as yes/no flips, laterality errors, and output-language mismatches. Our findings show that strong performance on English medical VQA benchmarks does not necessarily translate to robust behavior in Indonesian clinical contexts. We observe a performance gap of 8 to 25 percent between the English and Indonesian settings, depending on the evaluation metric. These results highlight the need for more inclusive multilingual evaluation of medical multimodal foundation models. The dataset is available at https://huggingface.co/datasets/Lab-IS/IndoRad-VQA.

Related papers

• Multilingual Hematology Visual Question Answering Dataset [8.690777023437052]
  WBCMor VQA is a clinically validated bilingual English, Urdu morphology aware VQA benchmark for leukemia and normal white blood cell analysis.<n>The benchmark is constructed using morphology-aware annotations from LeukemiaAttri and WBCAtt datasets and supported by a domain specific Urdu dictionary.<n>The proposed resource aims to facilitate the development of accessible and clinically relevant AI systems for multilingual healthcare environments.
  arXiv  Detail & Related papers  (2026-06-24T00:06:04Z)
• Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson's Disease [72.0406069194794]
  Speech representations often encode language-dependent structure that can confound dysarthria detection.<n>We propose a representation-level language shift that aligns source-language self-supervised speech representations with the target-language distribution.<n>We evaluate the approach on oral DDK recordings from Parkinson's disease speech datasets in Czech, German, and Spanish.
  arXiv  Detail & Related papers  (2026-03-23T17:23:39Z)
• MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations [4.451052650309736]
  We introduce MedErrBench, the first multilingual benchmark for error detection, localization, and correction.<n>Based on an expanded taxonomy of ten common error types, MedErrBench covers English, Arabic and Chinese.<n>Results reveal notable performance gaps, particularly in non-English settings.
  arXiv  Detail & Related papers  (2026-02-05T14:18:20Z)
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks [12.886024273517556]
  Large Language Models (LLMs) have become widely used in medical applications, such as clinical decision support, medical education, and medical question answering.<n>Yet, these models are often English-centric, limiting their robustness and reliability for linguistically diverse communities.<n>Recent work has highlighted discrepancies in performance in low-resource languages for various medical tasks, but the underlying causes remain poorly understood.
  arXiv  Detail & Related papers  (2026-02-05T06:52:46Z)
• ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages [3.3753770857559826]
  We conduct the first systematic audit of ASR performance on real world clinical interview data spanning Kannada, Hindi, and Indian English.<n>We compare leading models including Indic Whisper, Whisper, Sarvam, Google speech to text, Gemma3n, Omnilingual, Vaani, and Gemini.<n>Results reveal substantial variability across models and languages, with some systems performing competitively on Indian English but failing on code mixed or vernacular speech.
  arXiv  Detail & Related papers  (2025-11-30T06:37:40Z)
• SwasthLLM: a Unified Cross-Lingual, Multi-Task, and Meta-Learning Zero-Shot Framework for Medical Diagnosis Using Contrastive Representations [0.4077787659104315]
  SwasthLLM is a unified, zero-shot, cross-lingual, and multi-task learning framework for medical diagnosis.<n>It operates effectively across English, Hindi, and Bengali without requiring language-specific fine-tuning.<n>SwasthLLM achieves high diagnostic performance, with a test accuracy of 97.22% and an F1-score of 97.17% in supervised settings.
  arXiv  Detail & Related papers  (2025-09-24T21:20:49Z)
• MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering [11.575146661047368]
  We introduce MORQA, a new multilingual benchmark designed to assess the effectiveness of NLG evaluation metrics.<n>We benchmark both traditional metrics and large language model (LLM)-based evaluators, such as GPT-4 and Gemini.<n>Our results provide the first comprehensive, multilingual qualitative study of NLG evaluation in the medical domain.
  arXiv  Detail & Related papers  (2025-09-15T19:51:57Z)
• Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering [68.3400058037817]
  We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality.<n>We show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations.
  arXiv  Detail & Related papers  (2025-04-10T09:24:54Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• Delving Deeper into Cross-lingual Visual Question Answering [115.16614806717341]
  We show that simple modifications to the standard training setup can substantially reduce the transfer gap to monolingual English performance. We analyze cross-lingual VQA across different question types of varying complexity for different multilingual multimodal Transformers.
  arXiv  Detail & Related papers  (2022-02-15T18:22:18Z)
• CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark [51.38557174322772]
  We present the first Chinese Biomedical Language Understanding Evaluation benchmark. It is a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification. We report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling.
  arXiv  Detail & Related papers  (2021-06-15T12:25:30Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.