Related papers: What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations

What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations

URL: http://arxiv.org/abs/2409.02449v4
Date: Sat, 09 Nov 2024 06:37:01 GMT
Title: What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations
Authors: Kavya Manohar, Leena G Pillai, Elizabeth Sherly,
Abstract summary: We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta's MMS, Seamless, and Assembly AI's Conformer. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, are fundamentally flawed when applied to Indic scripts. We propose a shift towards developing text normalization routines that leverage native linguistic expertise.
Score: 0.0
License:
Abstract: This paper explores the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, with a particular focus on Indic language scripts. We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta's MMS, Seamless, and Assembly AI's Conformer, and their unintended consequences on performance metrics. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, by removing inconsistencies such as variations in spelling, punctuation, and special characters, are fundamentally flawed when applied to Indic scripts. Through empirical analysis using text similarity scores and in-depth linguistic examination, we demonstrate that these flaws lead to artificially improved performance metrics for Indic languages. We conclude by proposing a shift towards developing text normalization routines that leverage native linguistic expertise, ensuring more robust and accurate evaluations of multilingual ASR models.