Svarah: Evaluating English ASR Systems on Indian Accents

• URL: http://arxiv.org/abs/2305.15760v1
• Date: Thu, 25 May 2023 06:20:29 GMT
• Title: Svarah: Evaluating English ASR Systems on Indian Accents
• Authors: Tahir Javed, Sakshi Joshi, Vignesh Nagarajan, Sai Sundaresan, Janki Nawale, Abhigyan Raman, Kaushal Bhogale, Pratyush Kumar, Mitesh M. Khapra
• Abstract summary: We create Svarah, a benchmark that contains 9.6 hours of transcribed English audio from 117 speakers across 65 geographic locations throughout India. Svarah comprises both read speech and spontaneous conversational data, covering various domains, such as history, culture, tourism, etc., ensuring a diverse vocabulary. We evaluate 6 open source ASR models and 2 commercial ASR systems on Svarah and show that there is clear scope for improvement on Indian accents.
• Score: 12.197514367387692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: India is the second largest English-speaking country in the world with a speaker base of roughly 130 million. Thus, it is imperative that automatic speech recognition (ASR) systems for English should be evaluated on Indian accents. Unfortunately, Indian speakers find a very poor representation in existing English ASR benchmarks such as LibriSpeech, Switchboard, Speech Accent Archive, etc. In this work, we address this gap by creating Svarah, a benchmark that contains 9.6 hours of transcribed English audio from 117 speakers across 65 geographic locations throughout India, resulting in a diverse range of accents. Svarah comprises both read speech and spontaneous conversational data, covering various domains, such as history, culture, tourism, etc., ensuring a diverse vocabulary. We evaluate 6 open source ASR models and 2 commercial ASR systems on Svarah and show that there is clear scope for improvement on Indian accents. Svarah as well as all our code will be publicly available.

Related papers

• Accent conversion using discrete units with parallel data synthesized from controllable accented TTS [56.18382038512251]
  The goal of accent conversion (AC) is to convert speech accents while preserving content and speaker identity. Previous methods either required reference utterances during inference, did not preserve speaker identity well, or used one-to-one systems that could only be trained for each non-native accent. This paper presents a promising AC model that can convert many accents into native to overcome these issues.
  arXiv  Detail & Related papers  (2024-09-30T19:52:10Z)
• LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems [16.143694951047024]
  We create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin.
  arXiv  Detail & Related papers  (2024-08-21T08:51:00Z)
• Accented Speech Recognition With Accent-specific Codebooks [53.288874858671576]
  Speech accents pose a significant challenge to state-of-the-art automatic speech recognition (ASR) systems. Degradation in performance across underrepresented accents is a severe deterrent to the inclusive adoption of ASR. We propose a novel accent adaptation approach for end-to-end ASR systems using cross-attention with a trainable set of codebooks.
  arXiv  Detail & Related papers  (2023-10-24T16:10:58Z)
• A Deep Dive into the Disparity of Word Error Rates Across Thousands of NPTEL MOOC Videos [4.809236881780707]
  We describe the curation of a massive speech dataset of 8740 hours consisting of $sim9.8$K technical lectures in the English language along with their transcripts delivered by instructors representing various parts of Indian demography. We use the curated dataset to measure the existing disparity in YouTube Automatic Captions and OpenAI Whisper model performance across the diverse demographic traits of speakers in India.
  arXiv  Detail & Related papers  (2023-07-20T05:03:00Z)
• Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR [14.15737970309719]
  We show that IndicWhisper significantly improves on considered ASR systems on the Vistaar benchmark. IndicWhisper has the lowest WER in 39 out of the 59 benchmarks, with an average reduction of 4.1 WER. We open-source all datasets, code and models.
  arXiv  Detail & Related papers  (2023-05-24T17:46:03Z)
• An Investigation of Indian Native Language Phonemic Influences on L2 English Pronunciations [5.3956335232250385]
  Growing number of L2 English speakers in India reinforces need to study accents and L1-L2 interactions. We investigate the accents of Indian English (IE) speakers and report in detail our observations, both specific and common to all regions. We demonstrate the influence of 18 Indian languages on IE by comparing the native language pronunciations with IE pronunciations obtained jointly from existing literature studies and phonetically annotated speech of 80 speakers.
  arXiv  Detail & Related papers  (2022-12-19T07:41:39Z)
• A study on native American English speech recognition by Indian listeners with varying word familiarity level [62.14295630922855]
  We have three kinds of responses from each listener while they recognize an utterance. From these transcriptions, word error rate (WER) is calculated and used as a metric to evaluate the similarity between the recognized and the original sentences. Speaker nativity wise analysis shows that utterances from speakers of some nativity are more difficult to recognize by Indian listeners compared to few other nativities.
  arXiv  Detail & Related papers  (2021-12-08T07:43:38Z)
• Towards Building ASR Systems for the Next Billion Users [15.867823754118422]
  We make contributions towards building ASR systems for low resource languages from the Indian subcontinent. First, we curate 17,000 hours of raw speech data for 40 Indian languages. Using this raw speech data we pretrain several variants of wav2vec style models for 40 Indian languages.
  arXiv  Detail & Related papers  (2021-11-06T19:34:33Z)
• Multilingual and code-switching ASR challenges for low resource Indian languages [59.2906853285309]
  We focus on building multilingual and code-switching ASR systems through two different subtasks related to a total of seven Indian languages. We provide a total of 600 hours of transcribed speech data, comprising train and test sets, in these languages. We also provide a baseline recipe for both the tasks with a WER of 30.73% and 32.45% on the test sets of multilingual and code-switching subtasks, respectively.
  arXiv  Detail & Related papers  (2021-04-01T03:37:01Z)
• Arabic Speech Recognition by End-to-End, Modular Systems and Human [56.96327247226586]
  We perform a comprehensive benchmarking for end-to-end transformer ASR, modular HMM-DNN ASR, and human speech recognition. For ASR the end-to-end work led to 12.5%, 27.5%, 23.8% WER; a new performance milestone for the MGB2, MGB3, and MGB5 challenges respectively. Our results suggest that human performance in the Arabic language is still considerably better than the machine with an absolute WER gap of 3.6% on average.
  arXiv  Detail & Related papers  (2021-01-21T05:55:29Z)
• Black-box Adaptation of ASR for Accented Speech [52.63060669715216]
  We introduce the problem of adapting a black-box, cloud-based ASR system to speech from a target accent. We propose a novel coupling of an open-source accent-tuned local model with the black-box service. Our fine-grained merging algorithm is better at fixing accent errors than existing word-level combination strategies.
  arXiv  Detail & Related papers  (2020-06-24T07:07:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.