Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models

• URL: http://arxiv.org/abs/2607.17164v1
• Date: Sun, 19 Jul 2026 09:54:52 GMT
• Title: Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models
• Abstract summary: This paper presents a controlled, fine-tuned Whisper-based Assamese ASR system trained on the Mozilla Common Voice 24.0-Assamese corpus.<n>A hardware-aware optimized training pipeline is implemented for resource-constrained environments.
• Score: 1.4095958360608893
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Developing Automatic Speech Recognition (ASR) for morphologically rich, low-resource languages such as Assamese is challenging due to insufficient annotated speech data. The pretrained Whisper model performs poorly on Assamese speech recognition tasks. This paper presents a controlled, fine-tuned Whisper-based Assamese ASR system trained on the Mozilla Common Voice 24.0-Assamese corpus. A hardware-aware optimized training pipeline is implemented for resource-constrained environments, employing mixed-precision training and gradient accumulation on Tesla 4 Graphics Processing Units (T4 GPUs). The proposed fine-tuned model significantly outperformed the Zero-shot baseline, yielding Word Error Rate (WER), Character Error Rate (CER), Match Error Rate (MER), and Word Infomation Loss (WIL) of 43.17\%, 13.18\%, 43\%, and 64.81\%, respectively, achieving significant relative improvements of 78.26\%, 93.10\%, 57.0\%, and 35.19\% over the baseline. Semantic evaluation of the fine-tuned model also demonstrates notable improvement over a zero baseline, attaining Bilingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit ORdering (METEOR) scores of 30.81 and 0.5262, respectively. Additionally, the predicted hallucination rate and Real-Time Factor (RTF) are substantially improved by 96.70\% and 32.38\%, compared to the zero-shot baseline.

Related papers

• A Speech Corpus for Mizo Automatic Speech Recognition: Whisper and SraVaani 1.0 Fine-Tuning with Morphology-Aware Evaluation [7.015860050706781]
  The development included collecting 17.62 hours of speech data, curating it, and fine-tuning the Mizo ASR system with three Whisper multilingual models and with the SraVaani 1.0 Indic multilingual model.<n> Whisper-large-v3 achieved the lowest conventional WER (18.08%), while morphology-aware evaluation yielded a WER of 7.22%.<n>The results demonstrate that the Whisper model can achieve a substantially low WER, even when adapted to an unseen language.
  arXiv  Detail & Related papers  (2026-08-19T18:30:46Z)
• Spoken Language Identification with Pre-trained Models and Margin Loss [9.800066159312749]
  For the speaker-controlled spoken language identification task proposed in the TidyLang Challenge 2026, this paper proposes a language identification method based on pre-trained models and margin-based losses.<n>The proposed method adopts a pre-trained ECAPA-TDNN as the feature encoder and incorporates margin-based losses to enhance the discriminative ability of language representations.
  arXiv  Detail & Related papers  (2026-05-03T14:37:52Z)
• Enhancing ASR Performance in the Medical Domain for Dravidian Languages [4.188779570200089]
  This work proposes a novel confidence-aware training framework that integrates real and synthetic speech data.<n>A 5-gram KenLM language model is applied for post-decoding correction.<n>Results show that the hybrid confidence-aware approach with learnable weights substantially reduces recognition errors.
  arXiv  Detail & Related papers  (2026-04-10T09:41:26Z)
• Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation [0.0]
  Pashto is spoken by approximately 60--80 million people but has no published benchmarks for multilingual automatic speech recognition (ASR) on any shared public test set.<n>This paper reports the first reproducible multi-model evaluation on public Pashto data, covering zero-shot ASR, script-level failure, and cross-domain evaluation of fine-tuned models.
  arXiv  Detail & Related papers  (2026-04-06T11:23:42Z)
• DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation [111.94720088481614]
  Can multimodal generative models effectively produce content given dialectal textual input?<n>We construct a new large-scale benchmark spanning six common English dialects.<n>We design a general encoder-based mitigation strategy for multimodal generative models.
  arXiv  Detail & Related papers  (2025-10-16T17:56:55Z)
• One Whisper to Grade Them All [10.035434464829958]
  We present an efficient end-to-end approach for holistic Automatic Speaking Assessment (ASA) of multi-part second-language tests.<n>Our system's main novelty is the ability to process all four spoken responses with a single Whisper-small encoder.<n>This architecture removes the need for transcription and per-part models, cuts inference time, and makes ASA practical for large-scale Computer-Assisted Language Learning systems.
  arXiv  Detail & Related papers  (2025-07-23T20:31:40Z)
• SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition [4.210105570047471]
  This paper investigates the performance of various speech SSL models on dialectal Arabic (DA) and Arabic-English code-switched (CS) speech.<n>To address data scarcity, a modified audio-splicing approach is introduced to generate artificial CS speech data.<n>Fine-tuning an already fine-tuned SSL model results in an absolute improvement on Word Error Rate (WER) of 7.8% on Arabic and English CS benchmarks.
  arXiv  Detail & Related papers  (2025-06-27T11:42:43Z)
• Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking [68.77659513993507]
  We present a simple and effective N-best re-ranking approach to improve multilingual ASR accuracy. Our results show spoken language identification accuracy improvements of 8.7% and 6.1%, respectively, and word error rates which are 3.3% and 2.0% lower on these benchmarks.
  arXiv  Detail & Related papers  (2024-09-27T03:31:32Z)
• Text Injection for Neural Contextual Biasing [57.589903308622745]
  This work proposes contextual text injection (CTI) to enhance contextual ASR. CTI with 100 billion text sentences can achieve up to 43.3% relative WER reduction from a strong neural biasing model.
  arXiv  Detail & Related papers  (2024-06-05T04:20:17Z)
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition [50.93943755401025]
  We propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition. We design different auxiliary neural architectures focusing on learnable pre-trained feature enhancement. Our methods outperform existing ASR tuning architectures and their extension with self-supervised losses.
  arXiv  Detail & Related papers  (2023-01-19T02:37:56Z)
• Unsupervised Cross-lingual Representation Learning for Speech Recognition [63.85924123692923]
  XLSR learns cross-lingual speech representations by pretraining a single model from the raw waveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over masked latent speech representations. Experiments show that cross-lingual pretraining significantly outperforms monolingual pretraining.
  arXiv  Detail & Related papers  (2020-06-24T18:25:05Z)
• Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition [66.47000813920619]
  We propose a non-autoregressive end-to-end speech recognition system called LASO. Because of the non-autoregressive property, LASO predicts a textual token in the sequence without the dependence on other tokens. We conduct experiments on publicly available Chinese dataset AISHELL-1.
  arXiv  Detail & Related papers  (2020-05-11T04:45:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.