Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization

• URL: http://arxiv.org/abs/2605.25928v1
• Date: Mon, 25 May 2026 15:07:48 GMT
• Title: Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization
• Abstract summary: We describe the winning system for Task 2 of the KSAA-2026 Shared Task on Arabic Speech Dictation with Automatic Diacritization.<n>Our system fine-tunes CATT-Whisper, a character-level multimodal model combining a pretrained CATT text encoder with a frozen Whisper speech encoder.<n>The system achieves 23.26% WER on the primary leaderboard metric, placing 1st among all participants.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We describe the winning system for Task 2 of the KSAA-2026 Shared Task on Arabic Speech Dictation with Automatic Diacritization. The task requires producing fully diacritized Arabic text from speech audio and undiacritized transcripts, with only 2,327 training samples available and no external data permitted. Our system fine-tunes CATT-Whisper, a character-level multimodal model combining a pretrained CATT text encoder with a frozen Whisper speech encoder. The key to our approach is training regularization: R-Drop consistency regularization, Optuna-optimized hyperparameters with high weight decay, and Focal Loss. At inference, we average 200 stochastic forward passes across four model checkpoints using Monte Carlo Dropout at the softmax probability level. The system achieves 23.26% WER on the primary leaderboard metric (with case endings, including no-diacritic positions), placing 1st among all participants.

Related papers

• SEAR: Segment-Evidence-Aware Routing for Weak-to-Strong Multilingual Speech MCQ [0.0]
  This paper describes our system for Task2 of the second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge.<n>A language model converts timestamped ASR into coherent event spans, which are expanded by a boundary margin and cropped from the original recording.<n>A text-only probe partitions the data into weak, text-answerable items used for supervised fine-tuning and strong, audio-dependent items used for reinforcement learning.
  arXiv  Detail & Related papers  (2026-09-10T10:38:19Z)
• RosettaSpeech: Zero-Shot Speech-to-Speech Translation from Monolingual Data [30.27234062544891]
  This paper introduces RosettaSpeech, a novel and simplified framework for zero-shot speech-to-speech translation (S2ST)<n>While our method leverages the linguistic knowledge inherent in text-based NMT models, it strictly eliminates the need for parallel speech-to-speech pairs.<n>Our model uses text as an intermediate bridge during training but functions as a direct, end-to-end speech-to-speech model at inference.
  arXiv  Detail & Related papers  (2025-11-26T02:02:20Z)
• Abjad AI at NADI 2025: CATT-Whisper: Multimodal Diacritic Restoration Using Text and Speech Representations [1.1391158217994781]
  We tackle the Diacritic Restoration (DR) task for Arabic dialectal sentences using a multimodal approach.<n>We propose a model that represents the text modality using an encoder extracted from our own pre-trained model named CATT.<n>Our experiments show that the proposed approach achieves a word error rate (WER) of 0.25 and a character error rate (CER) of 0.9 on the development set.
  arXiv  Detail & Related papers  (2025-10-28T09:58:18Z)
• End-to-end Automatic Speech Recognition and Speech Translation: Integration of Speech Foundational Models and LLMs [0.3867363075280544]
  Speech Translation (ST) is a machine translation task that involves converting speech signals from one language to the corresponding text in another language.<n>This paper explores a combined end-to-end architecture of pre-trained speech encoders and Large Language Models (LLMs) for performing both Automatic Speech Recognition (ASR) and ST simultaneously.
  arXiv  Detail & Related papers  (2025-10-11T20:10:30Z)
• VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning [64.56272011710735]
  We propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the large language models (LLMs) backbone.<n>Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks.
  arXiv  Detail & Related papers  (2024-10-23T00:36:06Z)
• CMU's IWSLT 2024 Simultaneous Speech Translation System [80.15755988907506]
  This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder.
  arXiv  Detail & Related papers  (2024-08-14T10:44:51Z)
• Cascaded Cross-Modal Transformer for Audio-Textual Classification [30.643750999989233]
  We propose to harness the inherent value of multimodal representations by transcribing speech using automatic speech recognition (ASR) models. We thus obtain an audio-textual (multimodal) representation for each data sample. We were declared the winning solution in the Requests Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics Challenge.
  arXiv  Detail & Related papers  (2024-01-15T10:18:08Z)
• SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training [106.34112664893622]
  We propose a unified-modal speech-unit-text pre-training model, SpeechUT, to connect the representations of a speech encoder and a text decoder with a shared unit encoder. Our proposed SpeechUT is fine-tuned and evaluated on automatic speech recognition (ASR) and speech translation (ST) tasks.
  arXiv  Detail & Related papers  (2022-10-07T17:57:45Z)
• ON-TRAC Consortium Systems for the IWSLT 2022 Dialect and Low-resource Speech Translation Tasks [8.651248939672769]
  This paper describes the ON-TRAC Consortium translation systems developed for two challenge tracks featured in the Evaluation Campaign of IWSLT 2022: low-resource and dialect speech translation. We build an end-to-end model as our joint primary submission, and compare it against cascaded models that leverage a large fine-tuned wav2vec 2.0 model for ASR. Our results highlight that self-supervised models trained on smaller sets of target data are more effective to low-resource end-to-end ST fine-tuning, compared to large off-the-shelf models.
  arXiv  Detail & Related papers  (2022-05-04T10:36:57Z)
• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation [76.13334392868208]
  Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues. In this work, we explore self-supervised pre-training with unlabeled speech data and data augmentation to tackle this issue.
  arXiv  Detail & Related papers  (2022-04-06T17:59:22Z)
• Textless Speech-to-Speech Translation on Real Data [49.134208897722246]
  We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language. We tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data.
  arXiv  Detail & Related papers  (2021-12-15T18:56:35Z)
• The Sequence-to-Sequence Baseline for the Voice Conversion Challenge 2020: Cascading ASR and TTS [66.06385966689965]
  This paper presents the sequence-to-sequence (seq2seq) baseline system for the voice conversion challenge (VCC) 2020. We consider a naive approach for voice conversion (VC), which is to first transcribe the input speech with an automatic speech recognition (ASR) model. We revisit this method under a sequence-to-sequence (seq2seq) framework by utilizing ESPnet, an open-source end-to-end speech processing toolkit.
  arXiv  Detail & Related papers  (2020-10-06T02:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.