SVEva Fair: A Framework for Evaluating Fairness in Speaker Verification
- URL: http://arxiv.org/abs/2107.12049v1
- Date: Mon, 26 Jul 2021 09:15:46 GMT
- Title: SVEva Fair: A Framework for Evaluating Fairness in Speaker Verification
- Authors: Wiebke Toussaint and Aaron Yi Ding
- Abstract summary: Speaker verification is a form of biometric identification that gives access to voice assistants.
Due to a lack of fairness metrics, little is known about how model performance varies across subgroups.
We develop SVEva Fair, an accessible, actionable and model-agnostic framework for evaluating the fairness of speaker verification components.
- Score: 1.2437226707039446
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Despite the success of deep neural networks (DNNs) in enabling on-device
voice assistants, increasing evidence of bias and discrimination in machine
learning is raising the urgency of investigating the fairness of these systems.
Speaker verification is a form of biometric identification that gives access to
voice assistants. Due to a lack of fairness metrics and evaluation frameworks
that are appropriate for testing the fairness of speaker verification
components, little is known about how model performance varies across
subgroups, and what factors influence performance variation. To tackle this
emerging challenge, we design and develop SVEva Fair, an accessible, actionable
and model-agnostic framework for evaluating the fairness of speaker
verification components. The framework provides evaluation measures and
visualisations to interrogate model performance across speaker subgroups and
compare fairness between models. We demonstrate SVEva Fair in a case study with
end-to-end DNNs trained on the VoxCeleb datasets to reveal potential bias in
existing embedded speech recognition systems based on the demographic
attributes of speakers. Our evaluation shows that publicly accessible benchmark
models are not fair and consistently produce worse predictions for some
nationalities, and for female speakers of most nationalities. To pave the way
for fair and reliable embedded speaker verification, SVEva Fair has been
implemented as an open-source python library and can be integrated into the
embedded ML development pipeline to facilitate developers and researchers in
troubleshooting unreliable speaker verification performance, and selecting high
impact approaches for mitigating fairness challenges
Related papers
- FairLENS: Assessing Fairness in Law Enforcement Speech Recognition [37.75768315119143]
We propose a novel and adaptable evaluation method to examine the fairness disparity between different models.
We conducted fairness assessments on 1 open-source and 11 commercially available state-of-the-art ASR models.
arXiv Detail & Related papers (2024-05-21T19:23:40Z) - AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models [92.92233932921741]
We propose the AV-SUPERB benchmark that enables general-purpose evaluation of unimodal audio/visual and bimodal fusion representations.
We evaluate 5 recent self-supervised models and show that none of these models generalize to all tasks.
We show that representations may be improved with intermediate-task fine-tuning and audio event classification with AudioSet serves as a strong intermediate task.
arXiv Detail & Related papers (2023-09-19T17:35:16Z) - DualFair: Fair Representation Learning at Both Group and Individual
Levels via Contrastive Self-supervision [73.80009454050858]
This work presents a self-supervised model, called DualFair, that can debias sensitive attributes like gender and race from learned representations.
Our model jointly optimize for two fairness criteria - group fairness and counterfactual fairness.
arXiv Detail & Related papers (2023-03-15T07:13:54Z) - Design Guidelines for Inclusive Speaker Verification Evaluation Datasets [0.6015898117103067]
Speaker verification (SV) provides billions of voice-enabled devices with access control, and ensures the security of voice-driven technologies.
Current SV evaluation practices are insufficient for evaluating bias: they are over-simplified and aggregate users, not representative of real-life usage scenarios.
This paper proposes design guidelines for constructing SV evaluation datasets that address these short-comings.
arXiv Detail & Related papers (2022-04-05T15:28:26Z) - Bias in Automated Speaker Recognition [0.0]
We study bias in the machine learning development workflow of speaker verification, a voice biometric and core task in automated speaker recognition.
We show that bias exists at every development stage in the well-known VoxCeleb Speaker Recognition Challenge.
Most affected are female speakers and non-US nationalities, who experience significant performance degradation.
arXiv Detail & Related papers (2022-01-24T06:48:57Z) - Bootstrap Equilibrium and Probabilistic Speaker Representation Learning
for Self-supervised Speaker Verification [15.652180150706002]
We propose self-supervised speaker representation learning strategies.
In the front-end, we learn the speaker representations via the bootstrap training scheme with the uniformity regularization term.
In the back-end, the probabilistic speaker embeddings are estimated by maximizing the mutual likelihood score between the speech samples belonging to the same speaker.
arXiv Detail & Related papers (2021-12-16T14:55:44Z) - LDNet: Unified Listener Dependent Modeling in MOS Prediction for
Synthetic Speech [67.88748572167309]
We present LDNet, a unified framework for mean opinion score (MOS) prediction.
We propose two inference methods that provide more stable results and efficient computation.
arXiv Detail & Related papers (2021-10-18T08:52:31Z) - VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised
Speech Representation Disentanglement for One-shot Voice Conversion [54.29557210925752]
One-shot voice conversion can be effectively achieved by speech representation disentanglement.
We employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training.
Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations.
arXiv Detail & Related papers (2021-06-18T13:50:38Z) - Improving Fairness in Speaker Recognition [4.94706680113206]
We investigate the disparity in performance achieved by state-of-the-art deep speaker recognition systems.
We show that models trained with demographically-balanced training sets exhibit a fairer behavior on different groups, while still being accurate.
arXiv Detail & Related papers (2021-04-29T01:08:53Z) - Self-supervised Text-independent Speaker Verification using Prototypical
Momentum Contrastive Learning [58.14807331265752]
We show that better speaker embeddings can be learned by momentum contrastive learning.
We generalize the self-supervised framework to a semi-supervised scenario where only a small portion of the data is labeled.
arXiv Detail & Related papers (2020-12-13T23:23:39Z) - Deep Speaker Embeddings for Far-Field Speaker Recognition on Short
Utterances [53.063441357826484]
Speaker recognition systems based on deep speaker embeddings have achieved significant performance in controlled conditions.
Speaker verification on short utterances in uncontrolled noisy environment conditions is one of the most challenging and highly demanded tasks.
This paper presents approaches aimed to achieve two goals: a) improve the quality of far-field speaker verification systems in the presence of environmental noise, reverberation and b) reduce the system qualitydegradation for short utterances.
arXiv Detail & Related papers (2020-02-14T13:34:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.