ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages

• URL: http://arxiv.org/abs/2512.10967v1
• Date: Sun, 30 Nov 2025 06:37:40 GMT
• Title: ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages
• Abstract summary: We conduct the first systematic audit of ASR performance on real world clinical interview data spanning Kannada, Hindi, and Indian English.<n>We compare leading models including Indic Whisper, Whisper, Sarvam, Google speech to text, Gemma3n, Omnilingual, Vaani, and Gemini.<n>Results reveal substantial variability across models and languages, with some systems performing competitively on Indian English but failing on code mixed or vernacular speech.
• Score: 3.3753770857559826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic Speech Recognition (ASR) is increasingly used to document clinical encounters, yet its reliability in multilingual and demographically diverse Indian healthcare contexts remains largely unknown. In this study, we conduct the first systematic audit of ASR performance on real world clinical interview data spanning Kannada, Hindi, and Indian English, comparing leading models including Indic Whisper, Whisper, Sarvam, Google speech to text, Gemma3n, Omnilingual, Vaani, and Gemini. We evaluate transcription accuracy across languages, speakers, and demographic subgroups, with a particular focus on error patterns affecting patients vs. clinicians and gender based or intersectional disparities. Our results reveal substantial variability across models and languages, with some systems performing competitively on Indian English but failing on code mixed or vernacular speech. We also uncover systematic performance gaps tied to speaker role and gender, raising concerns about equitable deployment in clinical settings. By providing a comprehensive multilingual benchmark and fairness analysis, our work highlights the need for culturally and demographically inclusive ASR development for healthcare ecosystem in India.

Related papers

• Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data [15.334108772901311]
  Ambient clinical scribes (ACS) are being rapidly deployed at scale across Global South healthcare settings.<n>These ACS are primarily developed or distilled from models built and validated on Global North speech, languages and consultation styles.<n>Indian clinical encounters are brief, triadic, multilingual, code-mixed with low-resource languages.
  arXiv  Detail & Related papers  (2026-09-15T15:52:15Z)
• SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs [56.34354368047577]
  Existing safety evaluation datasets for large language models (LLMs) predominantly focus on English and Western contexts.<n>We introduce SurakshaEval, a novel safety benchmark composed of human-written prompts spanning real-world scenarios.<n>We benchmark a broad range of state-of-the-art LLMs on SurakshaEval, establish baseline safety performance, and identify recurring failure modes.
  arXiv  Detail & Related papers  (2026-08-08T02:03:33Z)
• SamaVaani: Auditing and Debiasing Multilingual Clinical ASR for Indian Languages [4.7356929645399095]
  We conduct the systematic audit of Automatic Speech Recognition (ASR) performance on real-world psychiatric interview data spanning Kannada, Hindi and Indian English.<n>Our results reveal substantial variability across models and languages, with some systems performing competitively in Indian English but failing in regional speech.<n>We propose SamaVaani, a unified debiasing technique that simultaneously improves ASR performance and improves fairness across demographic groups.
  arXiv  Detail & Related papers  (2026-06-25T11:34:07Z)
• Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study [2.8292841621378844]
  We introduce IndoRad-VQA, an Indonesian adaptation of VQA-RAD, to assess whether medical VLMs retain radiology reasoning ability when questions are asked in Bahasa Indonesia.<n>We evaluate general-purpose, Southeast Asian multilingual, and medical-specific VLMs under English and prompting settings.<n>Our findings show that strong performance on English medical VQA benchmarks does not necessarily translate to robust behavior in Indonesian clinical contexts.
  arXiv  Detail & Related papers  (2026-06-02T14:14:27Z)
• Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seeking [72.66384771245389]
  This work introduces HEALTHDIAL, a large-scale, multilingual, and multi-parallel dataset for developing and evaluating retrieval-augmented generation (RAG)-based spoken dialogue systems.<n>The dataset comprises 6,000 information-seeking dialogues (1,500 per language) grounded in trusted content from the World Health Organization (WHO) and 163 hours of user speech recorded from native speakers of diverse dialects.
  arXiv  Detail & Related papers  (2026-05-28T15:47:09Z)
• Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson's Disease [72.0406069194794]
  Speech representations often encode language-dependent structure that can confound dysarthria detection.<n>We propose a representation-level language shift that aligns source-language self-supervised speech representations with the target-language distribution.<n>We evaluate the approach on oral DDK recordings from Parkinson's disease speech datasets in Czech, German, and Spanish.
  arXiv  Detail & Related papers  (2026-03-23T17:23:39Z)
• MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations [4.451052650309736]
  We introduce MedErrBench, the first multilingual benchmark for error detection, localization, and correction.<n>Based on an expanded taxonomy of ten common error types, MedErrBench covers English, Arabic and Chinese.<n>Results reveal notable performance gaps, particularly in non-English settings.
  arXiv  Detail & Related papers  (2026-02-05T14:18:20Z)
• Cross-Lingual Mental Health Ontologies for Indian Languages: Bridging Patient Expression and Clinical Understanding through Explainable AI and Human-in-the-Loop Validation [0.2555114504478013]
  Mental health communication in India is linguistically fragmented, culturally diverse, and often underrepresented in clinical communication.<n>We propose a framework for cross-lingual building of mental health representations through graph-based methods that capture culturally embedded expressions of distress, align them across languages, and link them with clinical terminology.
  arXiv  Detail & Related papers  (2025-10-06T21:27:37Z)
• BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages [93.92804151830744]
  We present BRIGHTER, a collection of multi-labeled, emotion-annotated datasets in 28 different languages.<n>We highlight the challenges related to the data collection and annotation processes.<n>We show that the BRIGHTER datasets represent a meaningful step towards addressing the gap in text-based emotion recognition.
  arXiv  Detail & Related papers  (2025-02-17T15:39:50Z)
• Advocating Character Error Rate for Multilingual ASR Evaluation [1.2597747768235845]
  We document the limitations of the word error rate (WER) as an evaluation metric and advocate for the character error rate (CER) as the primary metric. We show that CER avoids many of the challenges WER faces and exhibits greater consistency across writing systems. Our findings suggest that CER should be prioritized, or at least supplemented, in multilingual ASR evaluations to account for the varying linguistic characteristics of different languages.
  arXiv  Detail & Related papers  (2024-10-09T19:57:07Z)
• Zero-Shot Multi-Lingual Speaker Verification in Clinical Trials [4.231937382464348]
  In clinical trials, patients are assessed based on their speech data to detect and monitor cognitive and mental health disorders. We propose using these speech recordings to verify the identities of enrolled patients and identify and exclude the individuals who try to enroll multiple times in the same trial. We evaluate pre-trained TitaNet, ECAPA-TDNN, and SpeakerNet models by enrolling and testing with speech-impaired patients speaking English, German, Danish, Spanish, and Arabic languages.
  arXiv  Detail & Related papers  (2024-04-02T14:19:30Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• A Deep Dive into the Disparity of Word Error Rates Across Thousands of NPTEL MOOC Videos [4.809236881780707]
  We describe the curation of a massive speech dataset of 8740 hours consisting of $sim9.8$K technical lectures in the English language along with their transcripts delivered by instructors representing various parts of Indian demography. We use the curated dataset to measure the existing disparity in YouTube Automatic Captions and OpenAI Whisper model performance across the diverse demographic traits of speakers in India.
  arXiv  Detail & Related papers  (2023-07-20T05:03:00Z)
• Harnessing Cross-lingual Features to Improve Cognate Detection for Low-resource Languages [50.82410844837726]
  We demonstrate the use of cross-lingual word embeddings for detecting cognates among fourteen Indian languages. We evaluate our methods to detect cognates on a challenging dataset of twelve Indian languages. We observe an improvement of up to 18% points, in terms of F-score, for cognate detection.
  arXiv  Detail & Related papers  (2021-12-16T11:17:58Z)
• CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark [51.38557174322772]
  We present the first Chinese Biomedical Language Understanding Evaluation benchmark. It is a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification. We report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling.
  arXiv  Detail & Related papers  (2021-06-15T12:25:30Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• NUVA: A Naming Utterance Verifier for Aphasia Treatment [49.114436579008476]
  Assessment of speech performance using picture naming tasks is a key method for both diagnosis and monitoring of responses to treatment interventions by people with aphasia (PWA) Here we present NUVA, an utterance verification system incorporating a deep learning element that classifies 'correct' versus'incorrect' naming attempts from aphasic stroke patients. When tested on eight native British-English speaking PWA the system's performance accuracy ranged between 83.6% to 93.6%, with a 10-fold cross-validation mean of 89.5%.
  arXiv  Detail & Related papers  (2021-02-10T13:00:29Z)
• Vyaktitv: A Multimodal Peer-to-Peer Hindi Conversations based Dataset for Personality Assessment [50.15466026089435]
  We present a novel peer-to-peer Hindi conversation dataset- Vyaktitv. It consists of high-quality audio and video recordings of the participants, with Hinglish textual transcriptions for each conversation. The dataset also contains a rich set of socio-demographic features, like income, cultural orientation, amongst several others, for all the participants.
  arXiv  Detail & Related papers  (2020-08-31T17:44:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.