ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody

• URL: http://arxiv.org/abs/2603.18024v1
• Date: Thu, 05 Mar 2026 14:01:42 GMT
• Title: ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody
• Abstract summary: ProKWS is a novel framework integrating fine-grained phoneme learning with personalized prosody modeling.<n> Experiments show ProKWS delivers highly competitive performance, comparable to state-of-the-art models on standard benchmarks.
• Score: 9.903404890572086
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Current keyword spotting systems primarily use phoneme-level matching to distinguish confusable words but ignore user-specific pronunciation traits like prosody (intonation, stress, rhythm). This paper presents ProKWS, a novel framework integrating fine-grained phoneme learning with personalized prosody modeling. We design a dual-stream encoder where one stream derives robust phonemic representations through contrastive learning, while the other extracts speaker-specific prosodic patterns. A collaborative fusion module dynamically combines phonemic and prosodic information, enhancing adaptability across acoustic environments. Experiments show ProKWS delivers highly competitive performance, comparable to state-of-the-art models on standard benchmarks and demonstrates strong robustness for personalized keywords with tone and intent variations.

Related papers

• INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval [53.79335341504602]
  INSPIRE is the first benchmark for instruction-aware speech retrieval.<n>We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models.
  arXiv  Detail & Related papers  (2026-08-17T07:35:18Z)
• Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors [10.603967327834214]
  Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision.<n>Our method, ScenA, conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data.<n>We evaluate ScenA on the CoVoMix2-Dialogue benchmark, showing that it outperforms existing multi-speaker systems on speaker-binding metrics.
  arXiv  Detail & Related papers  (2026-06-17T17:51:50Z)
• MuFFIN: Multifaceted Pronunciation Feedback Model with Interactive Hierarchical Neural Modeling [14.953695326450001]
  We introduce MuFFIN, a Multi-Faceted pronunciation Feedback model with an Interactive hierarchical Neural architecture.<n>To better capture the nuanced distinctions between phonemes in the feature space, a novel phoneme-contrastive ordinal regularization mechanism is put forward.<n>We design a simple yet effective training objective, which is specifically tailored to perturb the outputs of a phoneme with the phoneme-specific variations.
  arXiv  Detail & Related papers  (2025-10-06T15:54:55Z)
• Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering [53.2204901422631]
  Text2Lip is a viseme-centric framework that constructs an interpretable phonetic-visual bridge.<n>We show that Text2Lip outperforms existing approaches in semantic fidelity, visual realism, and modality robustness.
  arXiv  Detail & Related papers  (2025-08-04T12:50:22Z)
• Learning Speaker-Invariant Visual Features for Lipreading [54.670614643480505]
  Lipreading is a challenging cross-modal task that aims to convert visual lip movements into spoken text.<n>Existing lipreading methods often extract speaker-specific lip attributes that introduce spurious correlations between vision and text.<n>We introduce SIFLip, a speaker-invariant visual feature learning framework that disentangles speaker-specific attributes.
  arXiv  Detail & Related papers  (2025-06-09T09:16:14Z)
• Generative Adversarial Training for Text-to-Speech Synthesis Based on Raw Phonetic Input and Explicit Prosody Modelling [0.36868085124383626]
  We describe an end-to-end speech synthesis system that uses generative adversarial training. We train our Vocoder for raw phoneme-to-audio conversion, using explicit phonetic, pitch and duration modeling.
  arXiv  Detail & Related papers  (2023-10-14T18:15:51Z)
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition [57.446013973449645]
  This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is validated with experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF.
  arXiv  Detail & Related papers  (2023-10-02T12:02:07Z)
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations [53.926969174260705]
  We present a novel method for phoneme-level prosody control of F0 and duration using intuitive discrete labels. We propose an unsupervised prosodic clustering process which is used to discretize phoneme-level F0 and duration features from a multispeaker speech dataset.
  arXiv  Detail & Related papers  (2022-11-29T15:43:36Z)
• Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesis [8.603535906880937]
  Cross-speaker style transfer is crucial to the applications of multi-style and expressive speech synthesis at scale. Existing style transfer methods are still far behind real application needs. We propose a cross-speaker style transfer text-to-speech model with explicit prosody bottleneck.
  arXiv  Detail & Related papers  (2021-07-27T02:43:57Z)
• STEPs-RL: Speech-Text Entanglement for Phonetically Sound Representation Learning [2.28438857884398]
  We present a novel multi-modal deep neural network architecture that uses speech and text entanglement for learning spoken-word representations. STEPs-RL is trained in a supervised manner to predict the phonetic sequence of a target spoken-word. Latent representations produced by our model were able to predict the target phonetic sequences with an accuracy of 89.47%.
  arXiv  Detail & Related papers  (2020-11-23T13:29:16Z)
• Disentangled Speech Embeddings using Cross-modal Self-supervision [119.94362407747437]
  We develop a self-supervised learning objective that exploits the natural cross-modal synchrony between faces and audio in video. We construct a two-stream architecture which: (1) shares low-level features common to both representations; and (2) provides a natural mechanism for explicitly disentangling these factors.
  arXiv  Detail & Related papers  (2020-02-20T14:13:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.