Self-supervised Speech Comparison for L2 Phone, Rhythm, and Intonation Scoring

• URL: http://arxiv.org/abs/2607.13721v1
• Date: Wed, 15 Jul 2026 11:37:23 GMT
• Title: Self-supervised Speech Comparison for L2 Phone, Rhythm, and Intonation Scoring
• Abstract summary: We investigate whether DTW over self-supervised WavLM representations can provide a text-free framework for assessing phonetic accuracy, rhythm, and intonation in English and Japanese L2 speech.<n>Results show that a basic DTW-based approach that compares learner speech to native templates exceeds human agreement on holistic and sentence-level phonetic scoring.
• Score: 25.343214931194264
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: L2 speech assessment has traditionally focused on phonetic assessment, leaving the scoring of suprasegmental features such as rhythm and intonation underexplored. Moreover, assessment methods often require training with labeled L2 speech data, making them difficult to apply in low-resource settings. We investigate whether DTW over self-supervised WavLM representations can provide a text-free framework for assessing phonetic accuracy, rhythm, and intonation in English and Japanese L2 speech. Results show that a basic DTW-based approach that compares learner speech to native templates exceeds human agreement on holistic and sentence-level phonetic scoring. For rhythm, we introduce methods that measure the degree of warping in the DTW alignment path; our best method approaches human-level performance. For intonation, we combine DTW distance over prosodic residuals with pitch and intensity features, but performance remains more modest on some tasks. Our results point to self-supervised representations as a promising, text-free basis for multi-aspect pronunciation assessment.

Related papers

• Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities [8.300738063140129]
  This paper evaluates the zero-shot performance of Qwen2-Audio-7B-Instruct, an instruction-tuned speech-LLM, on 5,000 Speechocean762 utterances.<n>The model generates scores for accuracy, fluency, prosody, and completeness, showing strong agreement with human ratings within +-2 tolerance.
  arXiv  Detail & Related papers  (2026-01-20T15:48:38Z)
• Towards Unsupervised Speech Recognition at the Syllable-Level [95.54031547995874]
  We introduce a syllable-level UASR framework based on masked language modeling.<n>We generalize effectively to Mandarin, a language that has remained particularly difficult for prior methods.
  arXiv  Detail & Related papers  (2025-10-04T02:56:33Z)
• Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models [19.864555505996112]
  We propose two approaches to incorporate contextual paralinguistic information into model training.<n>Our implicit method boosts performance (LLM-judged) by 38.41% on a human-annotated QA benchmark, reaching 46.02% when combined with the explicit approach.
  arXiv  Detail & Related papers  (2025-08-10T10:03:30Z)
• Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM [53.17360668423001]
  Overlapping Speech Detection (OSD) aims to identify regions where multiple speakers overlap in a conversation.<n>This work proposes a speaker-aware progressive OSD model that leverages a progressive training strategy to enhance the correlation between subtasks.<n> Experimental results show that the proposed method achieves state-of-the-art performance, with an F1 score of 82.76% on the AMI test set.
  arXiv  Detail & Related papers  (2025-05-29T07:47:48Z)
• Automatic Proficiency Assessment in L2 English Learners [51.652753736780205]
  Second language proficiency (L2) in English is usually perceptually evaluated by English teachers or expert evaluators.<n>This paper explores deep learning techniques for comprehensive L2 proficiency assessment, addressing both the speech signal and its correspondent transcription.
  arXiv  Detail & Related papers  (2025-05-05T12:36:03Z)
• Speechworthy Instruction-tuned Language Models [71.8586707840169]
  We show that both prompting and preference learning increase the speech-suitability of popular instruction-tuned LLMs. We share lexical, syntactical, and qualitative analyses to showcase how each method contributes to improving the speech-suitability of generated responses.
  arXiv  Detail & Related papers  (2024-09-23T02:34:42Z)
• Learning Speech Representation From Contrastive Token-Acoustic Pretraining [57.08426714676043]
  We propose "Contrastive Token-Acoustic Pretraining (CTAP)", which uses two encoders to bring phoneme and speech into a joint multimodal space. The proposed CTAP model is trained on 210k speech and phoneme pairs, achieving minimally-supervised TTS, VC, and ASR.
  arXiv  Detail & Related papers  (2023-09-01T12:35:43Z)
• Incorporating L2 Phonemes Using Articulatory Features for Robust Speech Recognition [2.8360662552057323]
  This study is on the efficient incorporation of the L2 phonemes, which in this work refer to Korean phonemes, through articulatory feature analysis. We employ the lattice-free maximum mutual information (LF-MMI) objective in an end-to-end manner, to train the acoustic model to align and predict one of multiple pronunciation candidates. Experimental results show that the proposed method improves ASR accuracy for Korean L2 speech by training solely on L1 speech data.
  arXiv  Detail & Related papers  (2023-06-05T01:55:33Z)
• Proficiency assessment of L2 spoken English using wav2vec 2.0 [3.4012007729454816]
  We use wav2vec 2.0 for assessing overall and individual aspects of proficiency on two small datasets. We find that this approach significantly outperforms the BERT-based baseline system trained on ASR and manual transcriptions used for comparison.
  arXiv  Detail & Related papers  (2022-10-24T12:36:49Z)
• FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention [66.77490220410249]
  We propose FragmentVC, in which the latent phonetic structure of the utterance from the source speaker is obtained from Wav2Vec 2.0. FragmentVC is able to extract fine-grained voice fragments from the target speaker utterance(s) and fuse them into the desired utterance. This approach is trained with reconstruction loss only without any disentanglement considerations between content and speaker information.
  arXiv  Detail & Related papers  (2020-10-27T09:21:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.