HARNESS: Lightweight Distilled Arabic Speech Foundation Models

• URL: http://arxiv.org/abs/2604.14186v1
• Date: Tue, 31 Mar 2026 16:56:33 GMT
• Title: HARNESS: Lightweight Distilled Arabic Speech Foundation Models
• Abstract summary: We present HArnESS, an Arabic-centric self-supervised speech model family trained from scratch with iterative self-distillation.<n>Our approach begins with a large bilingual Arabic-English teacher and progressively distills its knowledge into compressed student models.<n>Compared with HuBERT and XLS-R, HArnESS consistently improves performance on Arabic downstream tasks.
• Score: 15.43303906288125
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large self-supervised speech (SSL) models achieve strong downstream performance, but their size limits deployment in resource-constrained settings. We present HArnESS, an Arabic-centric self-supervised speech model family trained from scratch with iterative self-distillation, together with lightweight student variants that offer strong accuracy-efficiency trade-offs on Automatic Speech Recognition (ASR), Dialect Identification (DID), and Speech Emotion Recognition (SER). Our approach begins with a large bilingual Arabic-English teacher and progressively distills its knowledge into compressed student models while preserving Arabic-relevant acoustic and paralinguistic representations. We further study PCA-based compression of the teacher supervision signal to better match the capacity of shallow and thin students. Compared with HuBERT and XLS-R, HArnESS consistently improves performance on Arabic downstream tasks, while the compressed models remain competitive under substantial structural reduction. These results position HArnESS as a practical and accessible Arabic-centric SSL foundation for real-world speech applications.

Related papers

• GigaAM Multilingual: Foundation Model for Underrepresented Languages [63.406165255277195]
  We present GigaAM Multilingual, a Conformer encoder pre-trained on 2M hours of audio using a HuBERT-style objective.<n>We introduce a cluster-level data balancing strategy during pre-training and a domain-aware sampling method during fine-tuning to mitigate head-language dominance.
  arXiv  Detail & Related papers  (2026-07-11T15:48:03Z)
• Ara-Best-RQ: Multi Dialectal Arabic SSL [11.331527589218483]
  Ara-BEST-RQ is a family of self-supervised learning (SSL) models for Arabic speech processing.<n>We pre-train conformer-based BEST-RQ models up to 600M parameters.<n>Models are evaluated on dialect identification (DID) and automatic speech recognition (ASR) tasks.
  arXiv  Detail & Related papers  (2026-03-23T12:23:31Z)
• Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis [20.50741854108831]
  We present Habibi, a suite of specialized and unified text-to-speech models.<n>Our approach outperforms the leading commercial service in generation quality.<n>We create the first systematic benchmark for multi-dialect Arabic speech synthesis.
  arXiv  Detail & Related papers  (2026-01-20T10:02:11Z)
• HARNESS: Lightweight Distilled Arabic Speech Foundation Models [15.43303906288125]
  We introduce HArnESS, the first Arabic-centric self-supervised speech model family.<n>We train large bilingual HArnESS (HL) SSL models and distill knowledge into compressed student models (HS, HST)<n>We use low-rank approximation to further compact the teacher's discrete supervision into shallow, thin models.
  arXiv  Detail & Related papers  (2025-09-18T07:30:37Z)
• Munsit at NADI 2025 Shared Task 2: Pushing the Boundaries of Multidialectal Arabic ASR with Weakly Supervised Pretraining and Continual Supervised Fine-tuning [0.0]
  We present a scalable training pipeline that combines weakly supervised learning with supervised fine-tuning to develop a robust Arabic ASR model.<n>Our approach achieves state-of-the-art results, ranking first in the multi-dialectal Arabic ASR challenge.
  arXiv  Detail & Related papers  (2025-08-12T13:02:22Z)
• Advancing Arabic Speech Recognition Through Large-Scale Weakly Supervised Learning [0.0]
  We employ weakly supervised learning to train an Arabic ASR model using the Conformer architecture.<n>Our model is trained from scratch on 15,000 hours of weakly annotated speech data covering both Modern Standard Arabic (MSA) and Dialectal Arabic (DA)
  arXiv  Detail & Related papers  (2025-04-16T17:05:14Z)
• Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion [70.23624194206171]
  This paper addresses the need for democratizing large language models (LLM) in the Arab world.<n>One practical objective for an Arabic LLM is to utilize an Arabic-specific vocabulary for the tokenizer that could speed up decoding.<n>Inspired by the vocabulary learning during Second Language (Arabic) Acquisition for humans, the released AraLLaMA employs progressive vocabulary expansion.
  arXiv  Detail & Related papers  (2024-12-16T19:29:06Z)
• SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT [45.729377551100676]
  We show that a syllabic organization emerges in learning sentence-level representation of speech.<n>We propose a new benchmark task, Spoken Speech ABX, for evaluating sentence-level representation of speech.
  arXiv  Detail & Related papers  (2023-10-16T20:05:36Z)
• Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages [70.25418443146435]
  We propose an efficient modeling framework for cross-lingual named entity recognition in semi-structured text data. We employ two independent datasets of SMSs in English and Arabic, each carrying semi-structured banking transaction information. With access to only 30 labeled samples, our model can generalize the recognition of merchants, amounts, and other fields from English to Arabic.
  arXiv  Detail & Related papers  (2023-07-16T00:45:42Z)
• Self-Supervised Learning for speech recognition with Intermediate layer supervision [52.93758711230248]
  We propose Intermediate Layer Supervision for Self-Supervised Learning (ILS-SSL) ILS-SSL forces the model to concentrate on content information as much as possible by adding an additional SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set show that our method outperforms HuBERT significantly.
  arXiv  Detail & Related papers  (2021-12-16T10:45:05Z)
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing [102.45426364965887]
  We propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM is built based on the HuBERT framework, with an emphasis on both spoken content modeling and speaker identity preservation. We scale up the training dataset from 60k hours to 94k hours of public audio data, and optimize its training procedure for better representation extraction.
  arXiv  Detail & Related papers  (2021-10-26T17:55:19Z)
• Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models [55.82292352607321]
  Code-switching (CS) is common in daily conversations where more than one language is used within a sentence. This paper uses the recently successful self-supervised learning (SSL) methods to leverage many unlabeled speech data without CS.
  arXiv  Detail & Related papers  (2021-10-07T14:43:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.