Where Do Self-Supervised Speech Models Become Unfair?

• URL: http://arxiv.org/abs/2604.18249v1
• Date: Mon, 20 Apr 2026 13:27:12 GMT
• Title: Where Do Self-Supervised Speech Models Become Unfair?
• Abstract summary: Speech encoder models are known to model members of some speaker groups (SGs) better than others.<n>We present the first layerwise fairness analysis of pretrained self-supervised speech encoder models (S3Ms)<n>We find S3Ms produce embeddings biased against certain SGs for both tasks, starting at the very first latent layers.
• Score: 44.43436255222398
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speech encoder models are known to model members of some speaker groups (SGs) better than others. However, there has been little work in establishing why this occurs on a technological level. To our knowledge, we present the first layerwise fairness analysis of pretrained self-supervised speech encoder models (S3Ms), probing each embedding layer for speaker identification (SID) automatic speech recognition (ASR). We find S3Ms produce embeddings biased against certain SGs for both tasks, starting at the very first latent layers. Furthermore, we find opposite patterns of layerwise bias for SID vs ASR for all models in our study: SID bias is minimized in layers that minimize overall SID error; on the other hand, ASR bias is maximized in layers that minimize overall ASR error. The inverse bias/error relationship for ASR is unaffected when probing S3Ms that are finetuned for ASR, suggesting SG-level bias is established during pretraining and is difficult to remove.

Related papers

• Speaker Group Encoding in Self-supervised Speech Recognition Models [6.54974886755316]
  Self-supervised speech recognition models (S3Ms) learn about speaker groups (SGs)<n>We find that S3Ms encode information about several speaker group categories (SGCs)<n>We discuss how our findings could be beneficial in designing fairer ASR algorithms.
  arXiv  Detail & Related papers  (2026-06-09T10:01:58Z)
• Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models [22.960033047333024]
  Self-supervised speech models (S3Ms) achieve strong downstream performance, yet their learned representations remain poorly understood under natural and adversarial perturbations.<n>We present GRIDS, a framework using Local Intrinsic Dimensionality (LID) across layer-wise representations in WavLM and wav2vec 2.0.
  arXiv  Detail & Related papers  (2026-05-04T15:18:15Z)
• Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models [47.083353570613916]
  This paper proposes a framework typifying two types of error that can occur in modeling phonemes in ASR systems.<n>We find that training phoneme classification probes only on a single, typically disadvantaged SG, sometimes improves performance for that SG.<n>We conclude that both types of error are present in phoneme embeddings and both are candidate causes for SG-level unfairness in ASR.
  arXiv  Detail & Related papers  (2026-04-24T15:03:39Z)
• Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses [71.34350093068473]
  This paper introduces a new paradigm for generative error correction (GER) framework in audio-visual speech recognition (AVSR)<n>Our framework, DualHyp, empowers a large language model (LLM) to compose independent N-best hypotheses from separate automatic speech recognition (ASR) and visual speech recognition (VSR) models.<n>Our framework attains up to 57.7% error rate gain on the LRS2 benchmark over standard ASR baseline, contrary to single-stream GER approaches that achieve only 10% gain.
  arXiv  Detail & Related papers  (2025-10-15T08:27:16Z)
• ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction [20.04650481108717]
  This paper studies ASR error correction in the Chinese language.<n>To the best of our knowledge, it is the first Chinese ASR error correction benchmark.<n>Inspired by the recent advances in emphlarge language models (LLMs), we investigate how to harness the power of LLMs to correct ASR errors.
  arXiv  Detail & Related papers  (2024-12-04T06:52:10Z)
• Denoising LM: Pushing the Limits of Error Correction Models for Speech Recognition [21.516152600963775]
  Denoising LM (DLM) is a $textitscaled$ error correction model trained with vast amounts of synthetic data. DLM achieves 1.5% word error rate (WER) on $textittest-clean$ and 3.3% WER on $textittest-other$ on Librispeech.
  arXiv  Detail & Related papers  (2024-05-24T05:05:12Z)
• ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding [55.39105863825107]
  We propose Mutual Learning and Large-Margin Contrastive Learning (ML-LMCL) to improve automatic speech recognition (ASR) robustness. In fine-tuning, we apply mutual learning and train two SLU models on the manual transcripts and the ASR transcripts, respectively. Experiments on three datasets show that ML-LMCL outperforms existing models and achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-11-19T16:53:35Z)
• Multimodal Audio-textual Architecture for Robust Spoken Language Understanding [18.702076738332867]
  multimodal language understanding (MLU) module is proposed to mitigate SLU performance degradation caused by errors in the ASR transcript. Our model is evaluated on five tasks from three SLU datasets and robustness is tested using ASR transcripts from three ASR engines. Results show that the proposed approach effectively mitigates the ASR error propagation problem, surpassing the PLM models' performance across all datasets for the academic ASR engine.
  arXiv  Detail & Related papers  (2023-06-12T01:55:53Z)
• Directional ASR: A New Paradigm for E2E Multi-Speaker Speech Recognition with Source Localization [73.62550438861942]
  This paper proposes a new paradigm for handling far-field multi-speaker data in an end-to-end neural network manner, called directional automatic speech recognition (D-ASR) In D-ASR, the azimuth angle of the sources with respect to the microphone array is defined as a latent variable. This angle controls the quality of separation, which in turn determines the ASR performance.
  arXiv  Detail & Related papers  (2020-10-30T20:26:28Z)
• Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining [64.35907499990455]
  We propose a framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation.
  arXiv  Detail & Related papers  (2020-10-26T18:21:27Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.