When Multiple Scripts Matter: Evaluating ASR in Clinical Settings

• URL: http://arxiv.org/abs/2606.17826v1
• Date: Tue, 16 Jun 2026 11:53:21 GMT
• Title: When Multiple Scripts Matter: Evaluating ASR in Clinical Settings
• Abstract summary: We introduce MultiClin, a benchmark designed to evaluate robustness to multiscript variability.<n>Experiments across diverse ASR models show that multiscript-aware evaluation provides a fairer assessment of recognition quality.<n> script unification consistently yields the best ASR performance.
• Score: 34.04832340475104
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic speech recognition (ASR) in non-English clinical settings is challenged by multiscript variability, where the same term may appear in multiple valid orthographic forms. Conventional string-matching evaluation metrics often underestimate ASR performance by treating orthographic variants as errors. To address this issue, we introduce MultiClin, a clinical ASR benchmark designed to evaluate robustness to multiscript variability. Experiments across diverse ASR models show that multiscript-aware evaluation provides a fairer assessment of recognition quality than conventional single-reference evaluation. We further investigate the impact of script consistency during training and find that inconsistent script mappings increase orthographic uncertainty and hinder model convergence, with a balanced 50% mapping ratio producing the highest entropy. In contrast, script unification consistently yields the best ASR performance. Our dataset and code are publicly available at: https://github.com/aitrics-ronaldo/Interspeech_MultiClin.

Related papers

• Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection [52.986874008247554]
  We propose ReCAP, a language-free framework that replaces static anchors with input-conditioned visual prototypes.<n>ReCAP re-centers separated normal and abnormal prototypes for each image through a bounded gated modulation.<n>It achieves the best image-level AUROC on all zero-shot and 23 of 24 few-shot settings, and the best zero-shot pixel-level AUROC on all three segmentation datasets.
  arXiv  Detail & Related papers  (2026-08-01T04:46:55Z)
• SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation [0.8243795440723053]
  Script-Normalized WER (SN-WER) is a training-free, evaluation-only scoring method.<n>We evaluate SN-WER on 5 Indic languages, 2 datasets, and 3 ASR models.
  arXiv  Detail & Related papers  (2026-06-01T17:49:10Z)
• Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation [53.844308305341166]
  We propose textbfAgentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing.<n> Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors.
  arXiv  Detail & Related papers  (2026-05-28T06:23:31Z)
• Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition [52.63676763985825]
  We propose a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy.<n>We design an agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs.<n>We will release the code to facilitate future research in interactive and agentic ASR.
  arXiv  Detail & Related papers  (2026-04-10T09:02:42Z)
• UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities [70.79422099851506]
  We present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems.<n>UDE standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface.
  arXiv  Detail & Related papers  (2026-03-24T13:01:31Z)
• AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering [97.52852990265136]
  We introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models.<n>We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-21T07:35:36Z)
• Style-agnostic evaluation of ASR using multiple reference transcripts [0.3066137405373616]
  We attempt to mitigate some of these differences by performing style-agnostic evaluation of ASR systems.<n>We find that existing WER reports are likely significantly over-estimating the number of contentful errors made by state-of-the-art ASR systems.
  arXiv  Detail & Related papers  (2024-12-10T21:47:15Z)
• What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations [0.0]
  We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta's MMS, Seamless, and Assembly AI's Conformer. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, are fundamentally flawed when applied to Indic scripts. We propose a shift towards developing text normalization routines that leverage native linguistic expertise.
  arXiv  Detail & Related papers  (2024-09-04T05:08:23Z)
• Word-level confidence estimation for RNN transducers [7.12355127219356]
  We present a lightweight neural confidence model tailored for Automatic Speech Recognition (ASR) system with Recurrent Network Transducers (RNN-T) Compared to other existing approaches, our model utilizes: (a) the time information associated with recognized words, which reduces the computational complexity, and (b) a simple and elegant trick for mapping between sub-word and word sequences.
  arXiv  Detail & Related papers  (2021-09-28T18:38:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.