Content is What Remains: Invariant Speech Tokenization from Parallel Utterances

• URL: http://arxiv.org/abs/2607.19033v1
• Date: Tue, 21 Jul 2026 12:19:38 GMT
• Title: Content is What Remains: Invariant Speech Tokenization from Parallel Utterances
• Abstract summary: PINT fine-tunes an SSL encoder with alignment losses across parallel utterances and augmentations to distill this shared residual.<n>Unlike ASR text, PINT tokens preserve frame-level temporal grounding and serve as drop-in semantic targets for audio codecs.
• Score: 8.166364251367627
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete speech tokenizers aim to disentangle semantic from acoustic information, yet targets from self-supervised learning (SSL) models like HuBERT retain non-linguistic variation: speaker identity, prosody, and channel conditions leak into the tokens, inflating entropy. Our key insight is that when enough speakers utter the same words under varying conditions, linguistic content is the only shared factor. We propose PINT (Parallel INvariant Tokenization), which fine-tunes an SSL encoder with alignment losses across parallel utterances and augmentations to distill this shared residual. PINT collapses identical words onto consistent token sequences, drastically reducing conditional entropy. Unlike ASR text, PINT tokens preserve frame-level temporal grounding and serve as drop-in semantic targets for audio codecs. Experiments show a 98.7% relative reduction in speaker probe accuracy (93.1% to 1.2%), a 42% lower ABX error rate, and 27-30% lower LM perplexity versus baselines, confirming that the right invariance is key to efficient learning.

Related papers

• Not All Speech Is Intent: Adaptive Self-Correcting Inference Layer for Post-ASR False Wake-Up [2.497926557563177]
  False wake-up activations remain a persistent challenge in conversational AI.<n>We introduce the Feedback-Driven Adaptive Self-Correcting Inference Layer (ASCIL)<n>ASCIL re-evaluates wake-up intent before response generation by fusing acoustic embeddings, linguistic cues, device context, and patterns from past misclassifications.
  arXiv  Detail & Related papers  (2026-09-11T06:06:06Z)
• TokAN: Accent Normalization Using Self-Supervised Speech Tokens [34.77160718038654]
  TokAN is a token-based accent normalization framework.<n>An autoregressive encoder-decoder model performs token-to-token conversion.<n>A flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens.
  arXiv  Detail & Related papers  (2026-07-04T15:45:38Z)
• Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal [10.584132982823883]
  Training automated pronunciation assessment often relies on labeled learner errors or non-native corpora that are costly to collect.<n>We propose a lightweight framework trained only on native speech resources, operating unsupervised or lightly calibrated with a small set of scored utterances.
  arXiv  Detail & Related papers  (2026-06-18T08:04:16Z)
• PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers [13.355084498116193]
  PersonaGesture is a pipeline for single-reference co-speech gesture personalization of unseen speakers.<n>Two key components, Adaptive Style Infusion (ASI) and Implicit Distribution Rectification Rectification (IDR), separate temporal identity evidence from residual statistic correction.<n>Experiments show that separating denoising-time speaker memory from conservative post-generation moment correction improves unseen-speaker personalization over collapsed style codes.
  arXiv  Detail & Related papers  (2026-05-07T11:48:39Z)
• LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers [76.59130257385826]
  Intermediate merge residues in BPE vocabularies are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage.<n>We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens.<n>Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.
  arXiv  Detail & Related papers  (2026-02-04T16:19:05Z)
• StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs [54.229363096087866]
  Speech tokenizers are not robust to meaning-irrelevant acoustic perturbations.<n>This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal.<n>We introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism.
  arXiv  Detail & Related papers  (2025-09-26T11:32:51Z)
• Tokens with Meaning: A Hybrid Tokenization Approach for NLP [0.2826977330147589]
  Tokenization plays a pivotal role in natural language processing (NLP)<n>We introduce a hybrid tokenization framework that combines rule-based morphological analysis with statistical subword segmentation.<n>The method uses phono normalization, root-affix, and a novel algorithm that balances morpheme preservation with vocabulary efficiency.
  arXiv  Detail & Related papers  (2025-08-19T22:17:42Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR [74.38242498079627]
  Self-supervised learning (SSL) based discrete speech representations are highly compact and domain adaptable.<n>In this paper, SSL discrete speech features extracted from WavLM models are used as additional cross-utterance acoustic context features in Zipformer-Transducer ASR systems.
  arXiv  Detail & Related papers  (2024-09-13T13:01:09Z)
• Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation [89.88214896713846]
  STA score considers two critical factors: temporal redundancy and semantic importance. We apply the STA module to off-the-shelf video Transformers and Videowins. Results: Kinetics-400 and Something-Something V2 achieve 30% overshelf reduction with a negligible 0.2% accuracy drop.
  arXiv  Detail & Related papers  (2023-08-08T19:38:15Z)
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition [52.71604809100364]
  We propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets.
  arXiv  Detail & Related papers  (2021-10-11T00:08:48Z)
• Fast End-to-End Speech Recognition via a Non-Autoregressive Model and Cross-Modal Knowledge Transferring from BERT [72.93855288283059]
  We propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once) The model consists of an encoder, a decoder, and a position dependent summarizer (PDS)
  arXiv  Detail & Related papers  (2021-02-15T15:18:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.