Do speech foundation models really learn words?

• URL: http://arxiv.org/abs/2609.10434v1
• Date: Wed, 09 Sep 2026 16:47:59 GMT
• Title: Do speech foundation models really learn words?
• Abstract summary: Good discrimination of words may be explained by good encoding of word form (phonemes)<n>We show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content.
• Score: 4.9931954371955465
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.

Related papers

• Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding [23.312539472103566]
  We consider how we can map a written English keyword to spoken realisations of that word in Hindi.<n>Our work demonstrates that cross-lingual word-to-speech mappings can be learned directly from visual grounding.
  arXiv  Detail & Related papers  (2026-08-27T10:22:07Z)
• Speech Codec Probing from Semantic and Phonetic Perspectives [49.01048570474675]
  Speech tokenizers are essential for connecting speech to large language models (LLMs) in multimodal systems.<n> emerging evidence suggests that what is termed "semantic" in speech representations does not align with text-derived semantics.
  arXiv  Detail & Related papers  (2026-03-11T03:32:25Z)
• The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoders [1.5566524830295307]
  We show that visual grounding affects model-internal representations of words.<n>We also find different effects in speech- vs. text-based language encoders.<n>Our findings could usefully inform the development of more efficient methods to enrich speech-based models with visually-informed semantics.
  arXiv  Detail & Related papers  (2025-09-19T10:16:58Z)
• ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models [70.56468982313834]
  We propose ProsodyLM, which introduces a simple tokenization scheme amenable to learning prosody.<n>We find that ProsodyLM can learn surprisingly diverse emerging prosody processing capabilities through pre-training alone.
  arXiv  Detail & Related papers  (2025-07-27T00:59:01Z)
• Languages in Multilingual Speech Foundation Models Align Both Phonetically and Semantically [58.019484208091534]
  Cross-lingual alignment in pretrained language models (LMs) has enabled efficient transfer in text-based LMs.<n>It remains an open question whether findings and methods from text-based cross-lingual alignment apply to speech.
  arXiv  Detail & Related papers  (2025-05-26T07:21:20Z)
• SememeASR: Boosting Performance of End-to-End Speech Recognition against Domain and Long-Tailed Data Shift with Sememe Semantic Knowledge [58.979490858061745]
  We introduce sememe-based semantic knowledge information to speech recognition. Our experiments show that sememe information can improve the effectiveness of speech recognition. In addition, our further experiments show that sememe knowledge can improve the model's recognition of long-tailed data.
  arXiv  Detail & Related papers  (2023-09-04T08:35:05Z)
• Self-Supervised Speech Representation Learning: A Review [105.1545308184483]
  Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. This review presents approaches for self-supervised speech representation learning and their connection to other research areas.
  arXiv  Detail & Related papers  (2022-05-21T16:52:57Z)
• Protecting gender and identity with disentangled speech representations [49.00162808063399]
  We show that protecting gender information in speech is more effective than modelling speaker-identity information. We present a novel way to encode gender information and disentangle two sensitive biometric identifiers.
  arXiv  Detail & Related papers  (2021-04-22T13:31:41Z)
• STEPs-RL: Speech-Text Entanglement for Phonetically Sound Representation Learning [2.28438857884398]
  We present a novel multi-modal deep neural network architecture that uses speech and text entanglement for learning spoken-word representations. STEPs-RL is trained in a supervised manner to predict the phonetic sequence of a target spoken-word. Latent representations produced by our model were able to predict the target phonetic sequences with an accuracy of 89.47%.
  arXiv  Detail & Related papers  (2020-11-23T13:29:16Z)
• End-to-End Spoken Language Understanding Without Full Transcripts [38.19173637496798]
  We develop end-to-end (E2E) spoken language understanding systems that directly convert speech input to semantic entities. We create two types of such speech-to-entities models, a CTC model and an attention-based encoder-decoder model. For our speech-to-entities experiments on the ATIS corpus, both the CTC and attention models showed impressive ability to skip non-entity words.
  arXiv  Detail & Related papers  (2020-09-30T01:54:13Z)
• Analyzing autoencoder-based acoustic word embeddings [37.78342106714364]
  Acoustic word embeddings (AWEs) are representations of words which encode their acoustic features. We analyze basic properties of AWE spaces learned by a sequence-to-sequence encoder-decoder model in six typologically diverse languages. AWEs exhibit a word onset bias, similar to patterns reported in various studies on human speech processing and lexical access.
  arXiv  Detail & Related papers  (2020-04-03T16:11:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.