Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

• URL: http://arxiv.org/abs/2607.26350v1
• Date: Tue, 28 Jul 2026 23:46:37 GMT
• Title: Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens
• Abstract summary: We present a systematic analysis of language sensitivity by varying either the NAC training language or the SSL pre-training language.<n>Results show that downstream performance is insensitive to the NAC training language but strongly dependent on the SSL pre-training language.<n>These findings suggest that a single NAC can be reused across languages, while aligning the SSL pre-training language with the target language is crucial.
• Score: 53.6274995041438
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural audio codecs (NACs) have become popular for obtaining speech representations as discrete tokens. Beyond compression, discrete tokens can be used to train self-supervised learning (SSL) models. Such models, referred to as codec-based SSL models, reduce data storage and computational cost, enabling scalable SSL pre-training. However, their language sensitivity remains unclear. When the language changes, codec-based SSL models may require retraining, which undermines their efficiency. In this paper, we present a systematic analysis of language sensitivity by varying either the NAC training language or the SSL pre-training language while keeping the other fixed. Experimental results show that downstream performance is insensitive to the NAC training language but strongly dependent on the SSL pre-training language. These findings suggest that a single NAC can be reused across languages, while aligning the SSL pre-training language with the target language is crucial.

Related papers

• Scaling Spoken Language Models with Syllabic Speech Tokenization [17.835120807367677]
  Spoken language models (SLMs) typically discretize speech into high-frame-rate tokens extracted from SSL speech models.<n>Recent SSL work introduces acoustic tokenization of speech at the syllable level.<n>Syllabic tokens can match or surpass the previous high-frame rate tokens while significantly cutting training and inference costs.
  arXiv  Detail & Related papers  (2025-09-30T17:59:09Z)
• Language Bias in Self-Supervised Learning For Automatic Speech Recognition [15.976590369684464]
  Self-supervised learning (SSL) is used in deep learning to train on large datasets without the need for expensive labelling of the data.<n>In this paper, we identify language-specificworks within XLS-R and test the performance of theseworks on a variety of different languages.
  arXiv  Detail & Related papers  (2025-01-31T17:16:45Z)
• Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning [69.77973092264338]
  We show that more powerful techniques can lead to more efficient pre-training, opening SSL to more research groups. We propose WavLabLM, which extends WavLM's joint prediction and denoising to 40k hours of data across 136 languages. We show that further efficiency can be achieved with a vanilla HuBERT Base model, which can maintain 94% of XLS-R's performance with only 3% of the data.
  arXiv  Detail & Related papers  (2023-09-26T23:55:57Z)
• SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge? [45.901645659694935]
  Self-supervised learning (SSL) for speech representation has been successfully applied in various downstream tasks. In this paper, we aim to clarify if speech SSL techniques can well capture linguistic knowledge.
  arXiv  Detail & Related papers  (2023-06-14T09:04:29Z)
• Audio Self-supervised Learning: A Survey [60.41768569891083]
  Self-Supervised Learning (SSL) targets at discovering general representations from large-scale data without requiring human annotations. Its success in the fields of computer vision and natural language processing have prompted its recent adoption into the field of audio and speech processing.
  arXiv  Detail & Related papers  (2022-03-02T15:58:29Z)
• Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models [55.82292352607321]
  Code-switching (CS) is common in daily conversations where more than one language is used within a sentence. This paper uses the recently successful self-supervised learning (SSL) methods to leverage many unlabeled speech data without CS.
  arXiv  Detail & Related papers  (2021-10-07T14:43:35Z)
• Continual-wav2vec2: an Application of Continual Learning for Self-Supervised Automatic Speech Recognition [0.23872611575805824]
  We present a method for continual learning of speech representations for multiple languages using self-supervised learning (SSL) Wav2vec models perform SSL on raw audio in a pretraining phase and then finetune on a small fraction of annotated data. We use ideas from continual learning to transfer knowledge from a previous task to speed up pretraining a new language task.
  arXiv  Detail & Related papers  (2021-07-26T10:39:03Z)
• LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech [63.84741259993937]
  Self-Supervised Learning (SSL) using huge unlabeled data has been successfully explored for image and natural language processing. Recent works also investigated SSL from speech. We propose LeBenchmark: a reproducible framework for assessing SSL from speech.
  arXiv  Detail & Related papers  (2021-04-23T08:27:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.