TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer

• URL: http://arxiv.org/abs/2405.02124v1
• Date: Fri, 3 May 2024 14:25:21 GMT
• Title: TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer
• Authors: Noé Tits, Prernna Bhatnagar, Thierry Dutoit,
• Abstract summary: We present a novel approach for text independent phone-to-audio alignment based on phoneme recognition, representation learning and knowledge transfer. We evaluate our model using synthetic native data from the TIMIT dataset and the SCRIBE dataset for American and British English. Our proposed model outperforms the state-of-the-art (charsiu) in statistical metrics and has applications in language learning and speech processing systems.
• Score: 3.9981390090442694
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In this paper, we present a novel approach for text independent phone-to-audio alignment based on phoneme recognition, representation learning and knowledge transfer. Our method leverages a self-supervised model (wav2vec2) fine-tuned for phoneme recognition using a Connectionist Temporal Classification (CTC) loss, a dimension reduction model and a frame-level phoneme classifier trained thanks to forced-alignment labels (using Montreal Forced Aligner) to produce multi-lingual phonetic representations, thus requiring minimal additional training. We evaluate our model using synthetic native data from the TIMIT dataset and the SCRIBE dataset for American and British English, respectively. Our proposed model outperforms the state-of-the-art (charsiu) in statistical metrics and has applications in language learning and speech processing systems. We leave experiments on other languages for future work but the design of the system makes it easily adaptable to other languages.

Related papers

• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• Learning Phonotactics from Linguistic Informants [54.086544221761486]
  Our model iteratively selects or synthesizes a data-point according to one of a range of information-theoretic policies. We find that the information-theoretic policies that our model uses to select items to query the informant achieve sample efficiency comparable to, or greater than, fully supervised approaches.
  arXiv  Detail & Related papers  (2024-05-08T00:18:56Z)
• Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning [31.26989690734889]
  In this paper, we present the development of an isometric NMT system using Reinforcement Learning (RL) To evaluate our models, we propose the Phoneme Count Compliance (PCC) score, which is a measure of length compliance. Our approach demonstrates a substantial improvement of approximately 36% in the PCC score compared to the state-of-the-art models when applied to English-Hindi language pairs.
  arXiv  Detail & Related papers  (2024-03-20T08:52:40Z)
• Few-Shot Cross-Lingual TTS Using Transferable Phoneme Embedding [55.989376102986654]
  This paper studies a transferable phoneme embedding framework that aims to deal with the cross-lingual text-to-speech problem under the few-shot setting. We propose a framework that consists of a phoneme-based TTS model and a codebook module to project phonemes from different languages into a learned latent space.
  arXiv  Detail & Related papers  (2022-06-27T11:24:40Z)
• Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition [71.49308685090324]
  This paper investigates the influence of different factors (i.e., model architecture, phonotactic model, type of speech representation) on phone recognition in an unknown language. We find that unique sounds, similar sounds, and tone languages remain a major challenge for phonetic inventory discovery.
  arXiv  Detail & Related papers  (2022-01-26T22:12:55Z)
• Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR [13.726142328715897]
  We present a method for cross-lingual training an ASR system using absolutely no transcribed training data from the target language. Our approach uses a novel application of a decipherment algorithm, which operates given only unpaired speech and text data from the target language.
  arXiv  Detail & Related papers  (2021-11-12T16:16:46Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Acoustics Based Intent Recognition Using Discovered Phonetic Units for Low Resource Languages [51.0542215642794]
  We propose a novel acoustics based intent recognition system that uses discovered phonetic units for intent classification. We present results for two languages families - Indic languages and Romance languages, for two different intent recognition tasks.
  arXiv  Detail & Related papers  (2020-11-07T00:35:31Z)
• LSTM Acoustic Models Learn to Align and Pronounce with Graphemes [22.453756228457017]
  We propose a grapheme-based speech recognizer that can be trained in a purely data-driven fashion. We show that the grapheme models are competitive in WER with their phoneme-output counterparts when trained on large datasets.
  arXiv  Detail & Related papers  (2020-08-13T21:38:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.