voice2mode: Phonation Mode Classification in Singing using Self-Supervised Speech Models

• URL: http://arxiv.org/abs/2602.13928v1
• Date: Sat, 14 Feb 2026 23:51:53 GMT
• Title: voice2mode: Phonation Mode Classification in Singing using Self-Supervised Speech Models
• Authors: Aju Ani Justus, Ruchit Agrawal, Sudarsana Reddy Kadiri, Shrikanth Narayanan,
• Abstract summary: We present voice2mode, a method for classification of four singing phonation modes (breathy, neutral, flow, and pressed) using embeddings extracted from large self-supervised speech models.<n>We show that HuBERT embeddings obtained from early layers yield the best result (95.7% accuracy with SVM), an absolute improvement of 12-15% over the best traditional baseline.
• Score: 44.76258866929522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present voice2mode, a method for classification of four singing phonation modes (breathy, neutral (modal), flow, and pressed) using embeddings extracted from large self-supervised speech models. Prior work on singing phonation has relied on handcrafted signal features or task-specific neural nets; this work evaluates the transferability of speech foundation models to singing phonation classification. voice2mode extracts layer-wise representations from HuBERT and two wav2vec2 variants, applies global temporal pooling, and classifies the pooled embeddings with lightweight classifiers (SVM, XGBoost). Experiments on a publicly available soprano dataset (763 sustained vowel recordings, four labels) show that foundation-model features substantially outperform conventional spectral baselines (spectrogram, mel-spectrogram, MFCC). HuBERT embeddings obtained from early layers yield the best result (~95.7% accuracy with SVM), an absolute improvement of ~12-15% over the best traditional baseline. We also show layer-wise behaviour: lower layers, which retain acoustic/phonetic detail, are more effective than top layers specialized for Automatic Speech Recognition (ASR).

Related papers

• Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer [59.57249127943914]
  We present a multilingual Audio-Visual Speech Recognition model incorporating several enhancements to improve performance and audio noise robustness. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets. Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%.
  arXiv  Detail & Related papers  (2024-03-14T01:16:32Z)
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition [57.446013973449645]
  This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is validated with experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF.
  arXiv  Detail & Related papers  (2023-10-02T12:02:07Z)
• Investigation of Self-supervised Pre-trained Models for Classification of Voice Quality from Speech and Neck Surface Accelerometer Signals [27.398425786898223]
  This study examines simultaneously-recorded speech and NSA signals in the classification of voice quality. The effectiveness of pre-trained models is compared in feature extraction between glottal source waveforms and raw signal waveforms for both speech and NSA inputs.
  arXiv  Detail & Related papers  (2023-08-06T23:16:54Z)
• Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representations [30.293081541301746]
  Self-supervised learning (SSL) speech models such as wav2vec and HuBERT have demonstrated state-of-the-art performance on automatic speech recognition. We argue that the problem is caused by the lack of disentangled representations and an utterance-level learning objective. Our models outperform the current best model, WavLM, on all utterance-level non-semantic tasks on the SUPERB benchmark with only 20% of labeled data.
  arXiv  Detail & Related papers  (2023-05-14T08:26:24Z)
• Self-supervised models of audio effectively explain human cortical responses to speech [71.57870452667369]
  We capitalize on the progress of self-supervised speech representation learning to create new state-of-the-art models of the human auditory system. We show that these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
  arXiv  Detail & Related papers  (2022-05-27T22:04:02Z)
• Self-Supervised Learning for speech recognition with Intermediate layer supervision [52.93758711230248]
  We propose Intermediate Layer Supervision for Self-Supervised Learning (ILS-SSL) ILS-SSL forces the model to concentrate on content information as much as possible by adding an additional SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set show that our method outperforms HuBERT significantly.
  arXiv  Detail & Related papers  (2021-12-16T10:45:05Z)
• Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding [6.278338686038089]
  phonetic posteriorgrams based methods have been quite popular in non-parallel singing voice conversion systems. Due to the lack of acoustic information in PPGs, style and naturalness of the converted singing voices are still limited. Our proposed model can significantly improve the naturalness of converted singing voices and the similarity with the target singer.
  arXiv  Detail & Related papers  (2021-10-10T10:27:20Z)
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation [51.37980448183019]
  We propose Audio ALBERT, a lite version of the self-supervised speech representation model. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.
  arXiv  Detail & Related papers  (2020-05-18T10:42:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.