Tone recognition in low-resource languages of North-East India: peeling the layers of SSL-based speech models

• URL: http://arxiv.org/abs/2506.03606v1
• Date: Wed, 04 Jun 2025 06:32:12 GMT
• Title: Tone recognition in low-resource languages of North-East India: peeling the layers of SSL-based speech models
• Abstract summary: This study explores the use of self-supervised learning (SSL) models for tone recognition in three low-resource languages from North Eastern India.<n>We evaluate four Wav2vec2.0 base models that were pre-trained on both tonal and non-tonal languages.<n>Our results show that tone recognition works best for Mizo and worst for Angami.
• Score: 8.3067762265238
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study explores the use of self-supervised learning (SSL) models for tone recognition in three low-resource languages from North Eastern India: Angami, Ao, and Mizo. We evaluate four Wav2vec2.0 base models that were pre-trained on both tonal and non-tonal languages. We analyze tone-wise performance across the layers for all three languages and compare the different models. Our results show that tone recognition works best for Mizo and worst for Angami. The middle layers of the SSL models are the most important for tone recognition, regardless of the pre-training language, i.e. tonal or non-tonal. We have also found that the tone inventory, tone types, and dialectal variations affect tone recognition. These findings provide useful insights into the strengths and weaknesses of SSL-based embeddings for tonal languages and highlight the potential for improving tone recognition in low-resource settings. The source code is available at GitHub 1 .

Related papers

• Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models [8.556170587475146]
  Self-supervised learning (SSL) has made significant advances in speech representation learning.<n> multilingual SSL models tend to underperform their monolingual counterparts on each individual language.<n>We introduce limited visual grounding into bilingual speech SSL models to reduce this performance gap.
  arXiv  Detail & Related papers  (2025-09-22T08:48:04Z)
• A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models [31.318688705100517]
  English and Mandarin wav2vec 2.0 models learn contextual representations of abstract suprasegmental categories. Models are better at representing features that exist in the language of their training data. HuBERT and WavLM learn similar representations to wav2vec 2.0, differing mainly in later layer performance.
  arXiv  Detail & Related papers  (2024-08-24T22:03:40Z)
• Mispronunciation detection using self-supervised speech representations [10.010024759851142]
  We study the use of SSL models for the task of mispronunciation detection for second language learners. We compare two downstream approaches: 1) training the model for phone recognition using native English data, and 2) training a model directly for the target task using non-native English data.
  arXiv  Detail & Related papers  (2023-07-30T21:20:58Z)
• SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge? [45.901645659694935]
  Self-supervised learning (SSL) for speech representation has been successfully applied in various downstream tasks. In this paper, we aim to clarify if speech SSL techniques can well capture linguistic knowledge.
  arXiv  Detail & Related papers  (2023-06-14T09:04:29Z)
• Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples [89.16814518860357]
  The objective of this work is to explore the learning of visually grounded speech models (VGS) from multilingual perspective. Our key contribution in this work is to leverage the power of a high-resource language in a bilingual visually grounded speech model to improve the performance of a low-resource language.
  arXiv  Detail & Related papers  (2023-03-30T16:34:10Z)
• Learning Cross-lingual Visual Speech Representations [108.68531445641769]
  Cross-lingual self-supervised visual representation learning has been a growing research topic in the last few years. We use the recently-proposed Raw Audio-Visual Speechs (RAVEn) framework to pre-train an audio-visual model with unlabelled data. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance.
  arXiv  Detail & Related papers  (2023-03-14T17:05:08Z)
• The Ability of Self-Supervised Speech Models for Audio Representations [53.19715501273934]
  Self-supervised learning (SSL) speech models have achieved unprecedented success in speech representation learning. We conduct extensive experiments on abundant speech and non-speech audio datasets to evaluate the representation ability of state-of-the-art SSL speech models. Results show that SSL speech models could extract meaningful features of a wide range of non-speech audio, while they may also fail on certain types of datasets.
  arXiv  Detail & Related papers  (2022-09-26T15:21:06Z)
• Self-Supervised Learning for speech recognition with Intermediate layer supervision [52.93758711230248]
  We propose Intermediate Layer Supervision for Self-Supervised Learning (ILS-SSL) ILS-SSL forces the model to concentrate on content information as much as possible by adding an additional SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set show that our method outperforms HuBERT significantly.
  arXiv  Detail & Related papers  (2021-12-16T10:45:05Z)
• Intent Classification Using Pre-Trained Embeddings For Low Resource Languages [67.40810139354028]
  Building Spoken Language Understanding systems that do not rely on language specific Automatic Speech Recognition is an important yet less explored problem in language processing. We present a comparative study aimed at employing a pre-trained acoustic model to perform Spoken Language Understanding in low resource scenarios. We perform experiments across three different languages: English, Sinhala, and Tamil each with different data sizes to simulate high, medium, and low resource scenarios.
  arXiv  Detail & Related papers  (2021-10-18T13:06:59Z)
• Continual-wav2vec2: an Application of Continual Learning for Self-Supervised Automatic Speech Recognition [0.23872611575805824]
  We present a method for continual learning of speech representations for multiple languages using self-supervised learning (SSL) Wav2vec models perform SSL on raw audio in a pretraining phase and then finetune on a small fraction of annotated data. We use ideas from continual learning to transfer knowledge from a previous task to speed up pretraining a new language task.
  arXiv  Detail & Related papers  (2021-07-26T10:39:03Z)
• Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition [80.446770909975]
  Linguistic knowledge is of great benefit to scene text recognition. How to effectively model linguistic rules in end-to-end deep networks remains a research challenge. We propose an autonomous, bidirectional and iterative ABINet for scene text recognition.
  arXiv  Detail & Related papers  (2021-03-11T06:47:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.