End-to-End real time tracking of children's reading with pointer network

• URL: http://arxiv.org/abs/2310.11486v1
• Date: Tue, 17 Oct 2023 16:12:18 GMT
• Title: End-to-End real time tracking of children's reading with pointer network
• Abstract summary: We show how a real time reading tracker can be built efficiently for children's voices. We employ a pointer network that directly learns to predict positions in the ground truth text conditioned on the streaming speech. Our best model can accurately track adult speech with 87.8% accuracy and the much harder and disfluent children's speech with 77.1% accuracy on CMU Kids data.
• Score: 11.362681035467121
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we explore how a real time reading tracker can be built efficiently for children's voices. While previously proposed reading trackers focused on ASR-based cascaded approaches, we propose a fully end-to-end model making it less prone to lags in voice tracking. We employ a pointer network that directly learns to predict positions in the ground truth text conditioned on the streaming speech. To train this pointer network, we generate ground truth training signals by using forced alignment between the read speech and the text being read on the training set. Exploring different forced alignment models, we find a neural attention based model is at least as close in alignment accuracy to the Montreal Forced Aligner, but surprisingly is a better training signal for the pointer network. Our results are reported on one adult speech data (TIMIT) and two children's speech datasets (CMU Kids and Reading Races). Our best model can accurately track adult speech with 87.8% accuracy and the much harder and disfluent children's speech with 77.1% accuracy on CMU Kids data and a 65.3% accuracy on the Reading Races dataset.

Related papers

• Edge Phoneme Recognition for Children's Speech through Age-Aware Training [0.0]
  We train a lightweight model to predict the age of the learner, as well as the phoneme sequence.<n>This has enabled the creation of PhonemeTrainer, an application that can run on most modern cellular phones.
  arXiv  Detail & Related papers  (2026-08-10T20:20:56Z)
• SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision [25.71776883846138]
  SpidR is a self-supervised speech representation model that efficiently learns representations with highly accessible phonetic information.<n>It is trained on raw waveforms using a masked prediction objective combined with self-distillation and online clustering.<n>It outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream language modeling benchmarks.
  arXiv  Detail & Related papers  (2025-12-23T12:22:25Z)
• KidSpeak: A General Multi-purpose LLM for Kids' Speech Recognition and Screening [29.54910094759367]
  KidSpeak is a speech-enhanced Foundation Model capable of both generative and discriminative tasks specifically tailored to children's speech patterns.<n>We propose the Flexible and Automatic Speech Aligner (FASA) and leverage the method to construct high quality datasets for training and evaluation.<n>This novel alignment tool significantly improves the quality of aligned children's speech from noisy data, enhancing data quality by 13.6x compared to human annotations.
  arXiv  Detail & Related papers  (2025-12-01T00:19:37Z)
• Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnet [72.53502346791814]
  We compare flat-start training across datasets, SSL representations (WavLM, XEUS), and decoder architectures.<n> SSL representations are biased toward adult speech, with flat-start training on child speech mitigating these biases.<n>Age-related ASR and speaker verification analysis highlights the limitations of proprietary models.
  arXiv  Detail & Related papers  (2025-08-22T17:59:35Z)
• Self-Supervised Models for Phoneme Recognition: Applications in Children's Speech for Reading Learning [9.670752318129326]
  We first compare wav2vec 2.0, HuBERT and WavLM models adapted to phoneme recognition in French child speech. We then adapt it by unfreezing its transformer blocks during fine-tuning on child speech. We show that WavLM base+ is more robust to various reading tasks and noise levels.
  arXiv  Detail & Related papers  (2025-03-06T18:57:16Z)
• Personalized Speech Recognition for Children with Test-Time Adaptation [21.882608966462932]
  Off-the-shelf automatic speech recognition (ASR) models primarily pre-trained on adult data tend to generalize poorly to children's speech. We devised a novel ASR pipeline to apply unsupervised test-time adaptation (TTA) methods for child speech recognition. Our results show that ASR models adapted with TTA methods significantly outperform the unadapted off-the-shelf ASR baselines both on average and statistically across individual child speakers.
  arXiv  Detail & Related papers  (2024-09-19T21:40:07Z)
• Is Child-Directed Speech Effective Training Data for Language Models? [34.46268640655943]
  We train GPT-2 and RoBERTa models on 29M words of English child-directed speech. We test whether the global developmental ordering or the local discourse ordering of children's training data supports high performance relative to other datasets. These findings support the hypothesis that, rather than proceeding from better data, the child's learning algorithm is substantially more data-efficient than current language modeling techniques.
  arXiv  Detail & Related papers  (2024-08-07T08:18:51Z)
• SignSpeak: Open-Source Time Series Classification for ASL Translation [0.12499537119440243]
  We propose a low-cost, real-time ASL-to-speech translation glove and an exhaustive training dataset of sign language patterns. We benchmarked this dataset with supervised learning models, such as LSTMs, GRUs and Transformers, where our best model achieved 92% accuracy. Our open-source dataset, models and glove designs provide an accurate and efficient ASL translator while maintaining cost-effectiveness.
  arXiv  Detail & Related papers  (2024-06-27T17:58:54Z)
• Enhancing Sign Language Detection through Mediapipe and Convolutional Neural Networks (CNN) [3.192629447369627]
  This research combines MediaPipe and CNNs for the efficient and accurate interpretation of ASL dataset. The accuracy achieved by the model on ASL datasets is 99.12%. The system will have applications in the communication, education, and accessibility domains.
  arXiv  Detail & Related papers  (2024-06-06T04:05:12Z)
• REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR [54.64158282822995]
  We propose REBORN,Reinforcement-Learned Boundary with Iterative Training for Unsupervised ASR. ReBORN alternates between training a segmentation model that predicts the boundaries of the segmental structures in speech signals and training the phoneme prediction model, whose input is the speech feature segmented by the segmentation model, to predict a phoneme transcription. We conduct extensive experiments and find that under the same setting, REBORN outperforms all prior unsupervised ASR models on LibriSpeech, TIMIT, and five non-English languages in Multilingual LibriSpeech.
  arXiv  Detail & Related papers  (2024-02-06T13:26:19Z)
• Jointly Learning Visual and Auditory Speech Representations from Raw Data [108.68531445641769]
  RAVEn is a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our design is asymmetric w.r.t. driven by the inherent differences between video and audio. RAVEn surpasses all self-supervised methods on visual speech recognition.
  arXiv  Detail & Related papers  (2022-12-12T21:04:06Z)
• Nonwords Pronunciation Classification in Language Development Tests for Preschool Children [7.224391516694955]
  This work aims to automatically evaluate whether the language development of children is age-appropriate. In this work, the task is to determine whether spoken nonwords have been uttered correctly. We compare different approaches that are motivated to model specific language structures.
  arXiv  Detail & Related papers  (2022-06-16T10:19:47Z)
• A study on the efficacy of model pre-training in developing neural text-to-speech system [55.947807261757056]
  This study aims to understand better why and how model pre-training can positively contribute to TTS system performance. It is found that the TTS system could achieve comparable performance when the pre-training data is reduced to 1/8 of its original size.
  arXiv  Detail & Related papers  (2021-10-08T02:09:28Z)
• Generative Conversational Networks [67.13144697969501]
  We propose a framework called Generative Conversational Networks, in which conversational agents learn to generate their own labelled training data. We show an average improvement of 35% in intent detection and 21% in slot tagging over a baseline model trained from the seed data.
  arXiv  Detail & Related papers  (2021-06-15T23:19:37Z)
• Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining [64.35907499990455]
  We propose a framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation.
  arXiv  Detail & Related papers  (2020-10-26T18:21:27Z)
• Continuous speech separation: dataset and analysis [52.10378896407332]
  In natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. This paper describes a dataset and protocols for evaluating continuous speech separation algorithms.
  arXiv  Detail & Related papers  (2020-01-30T18:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.