SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings

• URL: http://arxiv.org/abs/2607.01238v1
• Date: Fri, 01 May 2026 17:46:18 GMT
• Title: SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings
• Abstract summary: We propose speaker-aware grapheme representation model that enriches characters with their precise acoustic realizations.<n>We demonstrate that our model reduces word error rates by half in extreme low-resource settings compared to standard grapheme-based models.
• Score: 23.920215321354465
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in speech synthesis have shifted from phoneme representations to direct grapheme modeling. While phonemes address the one-to-many mapping between text and acoustics, they rely on grapheme-to-phoneme (G2P) systems that fail to capture speaker-specific acoustic variation. Prior work demonstrates that grapheme-based models outperform phoneme-based systems at scale, but not in low-resource settings. In this paper, we propose SPARCLE, a speaker-aware grapheme representation model that enriches characters with their precise acoustic realizations. SPARCLE is trained with a contrastive objective to align graphemes with corresponding Wav2Vec2 acoustic representations while conditioned on speaker identity. The resulting model serves as a replacement to G2P systems for downstream text-to-speech (TTS) tasks. We demonstrate that SPARCLE improves generation quality, reducing word error rates by half in extreme low-resource settings compared to standard grapheme-based models.

Related papers

• POWSM: A Phonetic Open Whisper-Style Speech Foundation Model [50.73202227472358]
  POWSM is the first unified framework capable of jointly performing multiple phone-related tasks.<n>Our training data, code and models are released to foster open science.
  arXiv  Detail & Related papers  (2025-10-28T21:43:45Z)
• VALLR: Visual ASR Language Model for Lip Reading [28.561566996686484]
  Lip Reading, or Visual Automatic Speech Recognition, is a complex task requiring the interpretation of spoken language exclusively from visual cues.<n>We propose a novel two-stage, phoneme-centric framework for Visual Automatic Speech Recognition (V-ASR)<n>First, our model predicts a compact sequence of phonemes from visual inputs using a Video Transformer with a CTC head.<n>This phoneme output then serves as the input to a fine-tuned Large Language Model (LLM), which reconstructs coherent words and sentences.
  arXiv  Detail & Related papers  (2025-03-27T11:52:08Z)
• Multi-modal Adversarial Training for Zero-Shot Voice Cloning [9.823246184635103]
  We propose a Transformer encoder-decoder architecture to conditionally discriminate between real and generated speech features. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model and training on Libriheavy, a large multi-speaker dataset. Our model achieves improvements over the baseline in terms of speech quality and speaker similarity.
  arXiv  Detail & Related papers  (2024-08-28T16:30:41Z)
• Fine-tuning CLIP Text Encoders with Two-step Paraphrasing [83.3736789315201]
  We introduce a straightforward fine-tuning approach to enhance the representations of CLIP models for paraphrases. Our model, which we call ParaCLIP, exhibits significant improvements over baseline CLIP models across various tasks.
  arXiv  Detail & Related papers  (2024-02-23T06:11:50Z)
• High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models [56.00939852727501]
  Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations. Non-autoregressive framework enhances controllability, and duration diffusion model enables diversified prosodic expression.
  arXiv  Detail & Related papers  (2023-09-27T09:27:03Z)
• SoundChoice: Grapheme-to-Phoneme Models with Semantic Disambiguation [10.016862617549991]
  This paper proposes SoundChoice, a novel Grapheme-to-Phoneme (G2P) architecture that processes entire sentences rather than operating at the word level. SoundChoice achieves a Phoneme Error Rate (PER) of 2.65% on whole-sentence transcription using data from LibriSpeech and Wikipedia.
  arXiv  Detail & Related papers  (2022-07-27T01:14:59Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• Extended Graph Temporal Classification for Multi-Speaker End-to-End ASR [77.82653227783447]
  We propose an extension of GTC to model the posteriors of both labels and label transitions by a neural network. As an example application, we use the extended GTC (GTC-e) for the multi-speaker speech recognition task.
  arXiv  Detail & Related papers  (2022-03-01T05:02:02Z)
• Direct speech-to-speech translation with discrete units [64.19830539866072]
  We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We propose to predict the self-supervised discrete representations learned from an unlabeled speech corpus instead. When target text transcripts are available, we design a multitask learning framework with joint speech and text training that enables the model to generate dual mode output (speech and text) simultaneously in the same inference pass.
  arXiv  Detail & Related papers  (2021-07-12T17:40:43Z)
• LSTM Acoustic Models Learn to Align and Pronounce with Graphemes [22.453756228457017]
  We propose a grapheme-based speech recognizer that can be trained in a purely data-driven fashion. We show that the grapheme models are competitive in WER with their phoneme-output counterparts when trained on large datasets.
  arXiv  Detail & Related papers  (2020-08-13T21:38:36Z)
• A systematic comparison of grapheme-based vs. phoneme-based label units for encoder-decoder-attention models [42.761409598613845]
  We do a systematic comparison between grapheme- and phoneme-based output labels for an encoder-decoder-attention ASR model. Experiments performed on the Switchboard 300h and LibriSpeech benchmarks show that phoneme-based modeling is competitive to grapheme-based encoder-decoder-attention modeling.
  arXiv  Detail & Related papers  (2020-05-19T09:54:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.