Speaker Group Encoding in Self-supervised Speech Recognition Models

• URL: http://arxiv.org/abs/2606.10654v1
• Date: Tue, 09 Jun 2026 10:01:58 GMT
• Title: Speaker Group Encoding in Self-supervised Speech Recognition Models
• Abstract summary: Self-supervised speech recognition models (S3Ms) learn about speaker groups (SGs)<n>We find that S3Ms encode information about several speaker group categories (SGCs)<n>We discuss how our findings could be beneficial in designing fairer ASR algorithms.
• Score: 6.54974886755316
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: We investigate what self-supervised speech recognition models (S3Ms) learn about speaker groups (SGs). We examine several states of S3Ms: pretrained, finetuned on speaker identification (SID), finetuned on automatic speech recognition (ASR), and ASR-finetuned using a fairness enhancing algorithm. We find that S3Ms encode information about several speaker group categories (SGCs), including their gender, age, dialect, ethnicity, and whether they are a native speaker. We find that finetuning for SID amplifies certain SGCs, namely those whose variance is more phonetic in nature, though it does not amplify other SGCs, namely those whose variance is more semantic in nature. On the other hand, finetuning for ASR discards phonetically variant speaker group information (SGI) but retains semantically variant SGI. We find that ASR algorithms designed for fairness improvement change to what extent SGI is encoded in S3Ms; however, this is primarily true for for phonetically variant SGCs, and less true for semantically variant SGCs. We discuss how SGI is encoded by each layer, and identify subdimensions of embeddings responsible for encoding different SGCs. Finally, we discuss how our findings could be beneficial in designing fairer ASR algorithms.

Related papers

• Phone Segmentation and Recognition through Phonological Activation Mapping [76.99244720091627]
  Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately.<n>We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), and one only needs to steer them to solve both tasks.<n>We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality.
  arXiv  Detail & Related papers  (2026-07-10T01:05:30Z)
• Where Do Self-Supervised Speech Models Become Unfair? [44.43436255222398]
  Speech encoder models are known to model members of some speaker groups (SGs) better than others.<n>We present the first layerwise fairness analysis of pretrained self-supervised speech encoder models (S3Ms)<n>We find S3Ms produce embeddings biased against certain SGs for both tasks, starting at the very first latent layers.
  arXiv  Detail & Related papers  (2026-04-20T13:27:12Z)
• Frontend Token Enhancement for Token-Based Speech Recognition [50.35062963870211]
  Discretized representations of speech signals are efficient alternatives to continuous features for speech recognition applications.<n>In this work, we introduce a system that estimates clean speech tokens from noisy speech and evaluate it on an ASR backend using semantic tokens.<n>We consider four types of enhancement models based on their input/token domains: wave-to-wave, token-to-output, continuous SSL features-to-token, and wave-to-token.
  arXiv  Detail & Related papers  (2026-02-04T05:02:15Z)
• Accent-Invariant Automatic Speech Recognition via Saliency-Driven Spectrogram Masking [1.108292291257035]
  We propose an accent-invariant ASR framework that integrates accent and dialect classification into the recognition pipeline.<n>Our approach involves training a spectrogram-based classifier to capture accent-specific cues, masking the regions most influential to its predictions, and using the masked spectrograms for data augmentation.<n>For Persian, we introduce a newly collected dataset spanning multiple regional accents, establishing the first systematic benchmark for accent variation in Persian ASR.
  arXiv  Detail & Related papers  (2025-10-10T16:41:53Z)
• AdaCS: Adaptive Normalization for Enhanced Code-Switching ASR [1.8533128809847572]
  Intra-sentential code-switching is a significant challenge for Automatic Speech Recognition systems.<n>AdaCS is a normalization model integrates an adaptive bias attention module into encoder-decoder network.<n> Experiments show that AdaCS outperforms previous state-of-the-art method on Vietnamese CS ASR normalization.
  arXiv  Detail & Related papers  (2025-01-13T07:27:00Z)
• Self-Supervised Speech Representations are More Phonetic than Semantic [52.02626675137819]
  Self-supervised speech models (S3Ms) have become an effective backbone for speech applications. We seek a more fine-grained analysis of the word-level linguistic properties encoded in S3Ms. Our study reveals that S3M representations consistently and significantly exhibit more phonetic than semantic similarity.
  arXiv  Detail & Related papers  (2024-06-12T20:04:44Z)
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models [66.11184017840688]
  We introduce C3LLM, a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities. Our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model.
  arXiv  Detail & Related papers  (2024-05-25T09:10:12Z)
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition [57.446013973449645]
  This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is validated with experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF.
  arXiv  Detail & Related papers  (2023-10-02T12:02:07Z)
• Explanations for Automatic Speech Recognition [9.810810252231812]
  We provide an explanation for an ASR transcription as a subset of audio frames. We adapt existing explainable AI techniques from image classification-Statistical Fault Localisation(SFL) and Causal. We evaluate the quality of the explanations generated by the proposed techniques over three different ASR,Google API, the baseline model of Sphinx, Deepspeech and 100 audio samples from the Commonvoice dataset.
  arXiv  Detail & Related papers  (2023-02-27T11:09:19Z)
• Learning Decoupling Features Through Orthogonality Regularization [55.79910376189138]
  Keywords spotting (KWS) and speaker verification (SV) are two important tasks in speech applications. We develop a two-branch deep network (KWS branch and SV branch) with the same network structure. A novel decoupling feature learning method is proposed to push up the performance of KWS and SV simultaneously.
  arXiv  Detail & Related papers  (2022-03-31T03:18:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.