Towards Unsupervised Speech Recognition at the Syllable-Level

• URL: http://arxiv.org/abs/2510.03639v1
• Date: Sat, 04 Oct 2025 02:56:33 GMT
• Title: Towards Unsupervised Speech Recognition at the Syllable-Level
• Authors: Liming Wang, Junrui Ni, Kai-Wei Chang, Saurabhchand Bhati, David Harwath, Mark Hasegawa-Johnson, James R. Glass,
• Abstract summary: We introduce a syllable-level UASR framework based on masked language modeling.<n>We generalize effectively to Mandarin, a language that has remained particularly difficult for prior methods.
• Score: 95.54031547995874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training speech recognizers with unpaired speech and text -- known as unsupervised speech recognition (UASR) -- is a crucial step toward extending ASR to low-resource languages in the long-tail distribution and enabling multimodal learning from non-parallel data. However, existing approaches based on phones often rely on costly resources such as grapheme-to-phoneme converters (G2Ps) and struggle to generalize to languages with ambiguous phoneme boundaries due to training instability. In this paper, we address both challenges by introducing a syllable-level UASR framework based on masked language modeling, which avoids the need for G2P and the instability of GAN-based methods. Our approach achieves up to a 40\% relative reduction in character error rate (CER) on LibriSpeech and generalizes effectively to Mandarin, a language that has remained particularly difficult for prior methods. Code will be released upon acceptance.

Related papers

• WESR: Scaling and Evaluating Word-level Event-Speech Recognition [59.21814194620928]
  Speech conveys not only linguistic information but also rich non-verbal vocal events such as laughing and crying.<n>We develop a refined taxonomy of 21 vocal events, with a new categorization into discrete (standalone) versus continuous (mixed with speech) types.<n>Based on the refined taxonomy, we introduce WESR-Bench, an expert-annotated evaluation set (900+ utterances) with a novel position-aware protocol.
  arXiv  Detail & Related papers  (2026-01-08T02:23:21Z)
• Cross-Lingual Interleaving for Speech Language Models [29.477655980414273]
  Spoken Language Models (SLMs) aim to learn linguistic competence directly from speech using discrete units.<n>We present a cross-lingual interleaving method that mixes speech tokens across languages without textual supervision.
  arXiv  Detail & Related papers  (2025-12-01T16:48:05Z)
• Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximation [12.39451124683428]
  We propose a latent variable model based method, with phonemes being treated as discrete latent variables.<n>Based on a multilingual pre-trained S2P model, crosslingual experiments are conducted in Polish and Indonesian.<n>With only 10 minutes of phoneme supervision, the new method, JSA-SPG, achieves 5% error rate reductions.
  arXiv  Detail & Related papers  (2025-07-04T12:23:22Z)
• SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition [55.2480439325792]
  Speech Meta In-Context LEarning (SMILE) is an innovative framework that combines meta-learning with speech in-context learning (SICL)<n>We show that SMILE consistently outperforms baseline methods in training-free few-shot multilingual ASR tasks.
  arXiv  Detail & Related papers  (2024-09-16T16:04:16Z)
• Towards Unsupervised Speech Recognition Without Pronunciation Models [57.222729245842054]
  In this article, we tackle the challenge of developing ASR systems without paired speech and text corpora.<n>We experimentally demonstrate that an unsupervised speech recognizer can emerge from joint speech-to-speech and text-to-text masked token-infilling.<n>This innovative model surpasses the performance of previous unsupervised ASR models under the lexicon-free setting.
  arXiv  Detail & Related papers  (2024-06-12T16:30:58Z)
• Efficient Spoken Language Recognition via Multilabel Classification [53.662747523872305]
  We show that our models obtain competitive results while being orders of magnitude smaller and faster than current state-of-the-art methods. Our multilabel strategy is more robust to unseen non-target languages compared to multiclass classification.
  arXiv  Detail & Related papers  (2023-06-02T23:04:19Z)
• Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning [4.327558819000435]
  We propose a novel joint textual-phonetic pre-training approach for learning spoken language representations. Experimental results on spoken language understanding benchmarks, Fluent Speech Commands and SNIPS, show that the proposed approach significantly outperforms strong baseline models.
  arXiv  Detail & Related papers  (2021-04-21T05:19:13Z)
• Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognition [58.849768879796905]
  We propose Adapt-and-Adjust (A2), a transformer-based multi-task learning framework for end-to-end multilingual speech recognition. The A2 framework overcomes the long-tail problem via three techniques: (1) exploiting a pretrained multilingual language model (mBERT) to improve the performance of low-resource languages; (2) proposing dual adapters consisting of both language-specific and language-agnostic adaptation with minimal additional parameters; and (3) overcoming the class imbalance, either by imposing class priors in the loss during training or adjusting the logits of the softmax output during inference.
  arXiv  Detail & Related papers  (2020-12-03T03:46:16Z)
• Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition [12.354292498112347]
  We present further improvements over our previous work by using domain adversarial learning to train task models. Our proposed technique leads to reductions in Word Error Rates (WER) in monolingual and code-switched test sets across three language pairs.
  arXiv  Detail & Related papers  (2020-06-09T13:45:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.