Boosting keyword spotting through on-device learnable user speech characteristics

• URL: http://arxiv.org/abs/2403.07802v1
• Date: Tue, 12 Mar 2024 16:41:31 GMT
• Title: Boosting keyword spotting through on-device learnable user speech characteristics
• Abstract summary: Keywords spotting systems for always-on TinyML-constrained applications require on-site tuning. Adapting to the speech peculiarities of target users requires many in-domain samples, often unavailable in real-world scenarios. We propose a novel on-device learning architecture, composed of a pretrained backbone and a user-aware embedding learning the user's speech characteristics.
• Score: 16.632185274553485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Keyword spotting systems for always-on TinyML-constrained applications require on-site tuning to boost the accuracy of offline trained classifiers when deployed in unseen inference conditions. Adapting to the speech peculiarities of target users requires many in-domain samples, often unavailable in real-world scenarios. Furthermore, current on-device learning techniques rely on computationally intensive and memory-hungry backbone update schemes, unfit for always-on, battery-powered devices. In this work, we propose a novel on-device learning architecture, composed of a pretrained backbone and a user-aware embedding learning the user's speech characteristics. The so-generated features are fused and used to classify the input utterance. For domain shifts generated by unseen speakers, we measure error rate reductions of up to 19% from 30.1% to 24.3% based on the 35-class problem of the Google Speech Commands dataset, through the inexpensive update of the user projections. We moreover demonstrate the few-shot learning capabilities of our proposed architecture in sample- and class-scarce learning conditions. With 23.7 kparameters and 1 MFLOP per epoch required for on-device training, our system is feasible for TinyML applications aimed at battery-powered microcontrollers.

Related papers

• Not All Fallbacks Are Failures: Understanding and Recovering from Fallbacks in Mobile Voice Assistants [50.58363289248172]
  We study fallback handling in a deployed smartwatch-based voice assistant for general health support in everyday environments.<n>Our analysis is based on six months of real-world usage data from more than 500 users, yielding a dataset of 3,030 anonymized, naturally occurring fallback-triggering utterances.
  arXiv  Detail & Related papers  (2026-08-31T13:06:42Z)
• Versatile On-device Adaptation at the Edge by Unifying Few-shot, Zero-shot, Continual, and In-context Learning [0.05402395391469584]
  We introduce embedder-centric learning (ECL), a framework that unifies four different online learning scenarios.<n>ECL can be deployed on resource-constrained devices across four real-world use cases.<n>We present first hardware demonstrations of ZSL with semantic data and ICL operating at micro-to-milliwatt power budgets.
  arXiv  Detail & Related papers  (2026-07-31T12:37:48Z)
• Enhancing Few-shot Keyword Spotting Performance through Pre-Trained Self-supervised Speech Models [3.25590215530292]
  Keywords Spotting plays a critical role in enabling hands-free interaction for battery-powered edge devices.<n>We propose a training scheme that leverages self-supervised learning models for robust feature extraction, dimensionality reduction, and knowledge distillation.<n>We evaluate the proposed approach on the English portion of the Multilingual Spoken Words Corpus (MSWC) and the Google Speech Commands (GSC) datasets.
  arXiv  Detail & Related papers  (2025-06-21T11:39:11Z)
• POET: Prompt Offset Tuning for Continual Human Action Adaptation [61.63831623094721]
  We aim to provide users and developers with the capability to personalize their experience by adding new action classes to their device models continually. We formalize this as privacy-aware few-shot continual action recognition. We propose a novel-temporal learnable prompt tuning approach, and are the first to apply such prompt tuning to Graph Neural Networks.
  arXiv  Detail & Related papers  (2025-04-25T04:11:24Z)
• Spatial Audio Processing with Large Language Model on Wearable Devices [6.345647878712574]
  We present a novel system architecture that incorporates spatial speech understanding into large language models (LLMs) SING supports spatially-aware automatic speech recognition (ASR), achieving a mean error of $25.72circ$-a substantial improvement compared to the 88.52circ$ median error in existing work-with a word error rate (WER) of 5.3. SING also supports soundscaping, for example, inference how many people were talking and their directions, with up to 5 people and a median DoA error of 16$circ$.
  arXiv  Detail & Related papers  (2025-04-11T18:19:59Z)
• Comprehensive Multi-Modal Prototypes are Simple and Effective Classifiers for Vast-Vocabulary Object Detection [68.26282316080558]
  Current open-world detectors can recognize a broader range of vocabularies, despite being trained on limited categories. We introduce Prova, a prototype classifier for vast-vocabulary object detection.
  arXiv  Detail & Related papers  (2024-12-23T18:57:43Z)
• A Unified Multi-Task Learning Architecture for Hate Detection Leveraging User-Based Information [23.017068553977982]
  Hate speech, offensive language, aggression, racism, sexism, and other abusive language are common phenomena in social media. There is a need for Artificial Intelligence(AI)based intervention which can filter hate content at scale. This paper introduces a unique model that improves hate speech identification for the English language by utilising intra-user and inter-user-based information.
  arXiv  Detail & Related papers  (2024-11-11T10:37:11Z)
• A Multimodal Approach to Device-Directed Speech Detection with Large Language Models [41.37311266840156]
  We explore whether it is feasible to drop the requirement that users must begin each command with a trigger phrase. We train classifiers using only acoustic information obtained from the audio waveform. We take the decoder outputs of an automatic speech recognition system, such as 1-best hypotheses, as input features to a large language model.
  arXiv  Detail & Related papers  (2024-03-21T14:44:03Z)
• Few-Shot Open-Set Learning for On-Device Customization of KeyWord Spotting Systems [41.24728444810133]
  This paper investigates few-shot learning methods for open-set KWS classification by combining a deep feature encoder with a prototype-based classifier. With user-defined keywords from 10 classes of the Google Speech Command dataset, our study reports an accuracy of up to 76% in a 10-shot scenario.
  arXiv  Detail & Related papers  (2023-06-03T17:10:33Z)
• Continual Learning for On-Device Speech Recognition using Disentangled Conformers [54.32320258055716]
  We introduce a continual learning benchmark for speaker-specific domain adaptation derived from LibriVox audiobooks. We propose a novel compute-efficient continual learning algorithm called DisentangledCL. Our experiments show that the DisConformer models significantly outperform baselines on general ASR.
  arXiv  Detail & Related papers  (2022-12-02T18:58:51Z)
• Incremental Online Learning Algorithms Comparison for Gesture and Visual Smart Sensors [68.8204255655161]
  This paper compares four state-of-the-art algorithms in two real applications: gesture recognition based on accelerometer data and image classification. Our results confirm these systems' reliability and the feasibility of deploying them in tiny-memory MCUs.
  arXiv  Detail & Related papers  (2022-09-01T17:05:20Z)
• Few-Shot Cross-Lingual TTS Using Transferable Phoneme Embedding [55.989376102986654]
  This paper studies a transferable phoneme embedding framework that aims to deal with the cross-lingual text-to-speech problem under the few-shot setting. We propose a framework that consists of a phoneme-based TTS model and a codebook module to project phonemes from different languages into a learned latent space.
  arXiv  Detail & Related papers  (2022-06-27T11:24:40Z)
• Building a Noisy Audio Dataset to Evaluate Machine Learning Approaches for Automatic Speech Recognition Systems [0.0]
  This work aims to present the process of building a dataset of noisy audios, in a specific case of degenerated audios due to interference. We also present initial results of a classifier that uses such data for evaluation, indicating the benefits of using this dataset in the recognizer's training process.
  arXiv  Detail & Related papers  (2021-10-04T13:08:53Z)
• Mobility-Aware Cluster Federated Learning in Hierarchical Wireless Networks [81.83990083088345]
  We develop a theoretical model to characterize the hierarchical federated learning (HFL) algorithm in wireless networks. Our analysis proves that the learning performance of HFL deteriorates drastically with highly-mobile users. To circumvent these issues, we propose a mobility-aware cluster federated learning (MACFL) algorithm.
  arXiv  Detail & Related papers  (2021-08-20T10:46:58Z)
• Dynamic Acoustic Unit Augmentation With BPE-Dropout for Low-Resource End-to-End Speech Recognition [62.94773371761236]
  We consider building an effective end-to-end ASR system in low-resource setups with a high OOV rate. We propose a method of dynamic acoustic unit augmentation based on the BPE-dropout technique. Our monolingual Turkish Conformer established a competitive result with 22.2% character error rate (CER) and 38.9% word error rate (WER)
  arXiv  Detail & Related papers  (2021-03-12T10:10:13Z)
• A Study of Few-Shot Audio Classification [2.1989764549743476]
  Few-shot learning is a type of machine learning designed to enable the model to generalize to new classes with very few examples. We evaluate our model for speaker identification on the VoxCeleb dataset and ICSI Meeting Corpus, obtaining 5-shot 5-way accuracies of 93.5% and 54.0%, respectively. We also evaluate for activity classification from audio using few-shot subsets of the Kinetics600 dataset and AudioSet, both drawn from Youtube videos, obtaining 51.5% and 35.2% accuracy, respectively.
  arXiv  Detail & Related papers  (2020-12-02T22:19:16Z)
• Wake Word Detection with Alignment-Free Lattice-Free MMI [66.12175350462263]
  Always-on spoken language interfaces, e.g. personal digital assistants, rely on a wake word to start processing spoken input. We present novel methods to train a hybrid DNN/HMM wake word detection system from partially labeled training data. We evaluate our methods on two real data sets, showing 50%--90% reduction in false rejection rates at pre-specified false alarm rates over the best previously published figures.
  arXiv  Detail & Related papers  (2020-05-17T19:22:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.