ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modeling

• URL: http://arxiv.org/abs/2602.15537v1
• Date: Tue, 17 Feb 2026 12:38:45 GMT
• Title: ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modeling
• Authors: Nicol Visser, Simon Malan, Danel Slabbert, Herman Kamper,
• Abstract summary: Pure speech language models aim to learn language directly from raw audio without textual resources.<n>We propose ZeroSyl, a training-free method to extract syllable boundaries and embeddings directly from a frozen WavLM model.
• Score: 14.08454786805815
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pure speech language models aim to learn language directly from raw audio without textual resources. A key challenge is that discrete tokens from self-supervised speech encoders result in excessively long sequences, motivating recent work on syllable-like units. However, methods like Sylber and SyllableLM rely on intricate multi-stage training pipelines. We propose ZeroSyl, a simple training-free method to extract syllable boundaries and embeddings directly from a frozen WavLM model. Using L2 norms of features in WavLM's intermediate layers, ZeroSyl achieves competitive syllable segmentation performance. The resulting segments are mean-pooled, discretized using K-means, and used to train a language model. ZeroSyl outperforms prior syllabic tokenizers across lexical, syntactic, and narrative benchmarks. Scaling experiments show that while finer-grained units are beneficial for lexical tasks, our discovered syllabic units exhibit better scaling behavior for syntactic modeling.

Related papers

• WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation [27.32235541083431]
  WavSLM is a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook.<n>It achieves competitive performance on consistency benchmarks and speech generation while using fewer parameters, less training data, and supporting streaming inference.
  arXiv  Detail & Related papers  (2026-03-05T15:39:54Z)
• SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision [25.71776883846138]
  SpidR is a self-supervised speech representation model that efficiently learns representations with highly accessible phonetic information.<n>It is trained on raw waveforms using a masked prediction objective combined with self-distillation and online clustering.<n>It outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream language modeling benchmarks.
  arXiv  Detail & Related papers  (2025-12-23T12:22:25Z)
• Scaling Spoken Language Models with Syllabic Speech Tokenization [17.835120807367677]
  Spoken language models (SLMs) typically discretize speech into high-frame-rate tokens extracted from SSL speech models.<n>Recent SSL work introduces acoustic tokenization of speech at the syllable level.<n>Syllabic tokens can match or surpass the previous high-frame rate tokens while significantly cutting training and inference costs.
  arXiv  Detail & Related papers  (2025-09-30T17:59:09Z)
• ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models [70.56468982313834]
  We propose ProsodyLM, which introduces a simple tokenization scheme amenable to learning prosody.<n>We find that ProsodyLM can learn surprisingly diverse emerging prosody processing capabilities through pre-training alone.
  arXiv  Detail & Related papers  (2025-07-27T00:59:01Z)
• Sylber: Syllabic Embedding Representation of Speech from Raw Audio [25.703703711031178]
  We propose a new model, Sylber, that produces speech representations with clean and robust syllabic structure.<n>Specifically, we propose a self-supervised learning framework that bootstraps syllabic embeddings by distilling from its own initial unsupervised syllabic segmentation.<n>This results in a highly structured representation of speech features, offering three key benefits: 1) a fast, linear-time syllable segmentation algorithm, 2) efficient syllabic tokenization with an average of 4.27 tokens per second, and 3) novel phonological units suited for efficient spoken language modeling.
  arXiv  Detail & Related papers  (2024-10-09T17:59:04Z)
• SyllableLM: Learning Coarse Semantic Units for Speech Language Models [21.762112843104028]
  We introduce a controllable self-supervised technique to merge speech representations into coarser syllable-like units. Our method produces controllable-rate semantic units at as low as 5Hz and 60bps and SotA inc segmentation and clustering. SyllableLM achieves significant improvements in efficiency with a 30x reduction in training compute and a 4x wall-clock inference speedup.
  arXiv  Detail & Related papers  (2024-10-05T04:29:55Z)
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens [49.569695524535454]
  We propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis.
  arXiv  Detail & Related papers  (2024-07-07T15:16:19Z)
• Generative Spoken Language Model based on continuous word-sized audio tokens [52.081868603603844]
  We introduce a Generative Spoken Language Model based on word-size continuous-valued audio embeddings. The resulting model is the first generative language model based on word-size continuous embeddings.
  arXiv  Detail & Related papers  (2023-10-08T16:46:14Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Quark: Controllable Text Generation with Reinforced Unlearning [68.07749519374089]
  Large-scale language models often learn behaviors that are misaligned with user expectations. We introduce Quantized Reward Konditioning (Quark), an algorithm for optimizing a reward function that quantifies an (un)wanted property. For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods.
  arXiv  Detail & Related papers  (2022-05-26T21:11:51Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.