Investigation of Different Calibration Methods for Deep Speaker Embedding based Verification Systems

• URL: http://arxiv.org/abs/2203.15106v1
• Date: Mon, 28 Mar 2022 21:22:22 GMT
• Title: Investigation of Different Calibration Methods for Deep Speaker Embedding based Verification Systems
• Authors: Galina Lavrentyeva, Sergey Novoselov, Andrey Shulipa, Marina Volkova, Aleksandr Kozlov
• Abstract summary: This paper presents an investigation over several methods of score calibration for deep speaker embedding extractors. An additional focus of this research is to estimate the impact of score normalization on the calibration performance of the system.
• Score: 66.61691401921296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep speaker embedding extractors have already become new state-of-the-art systems in the speaker verification field. However, the problem of verification score calibration for such systems often remains out of focus. An irrelevant score calibration leads to serious issues, especially in the case of unknown acoustic conditions, even if we use a strong speaker verification system in terms of threshold-free metrics. This paper presents an investigation over several methods of score calibration: a classical approach based on the logistic regression model; the recently presented magnitude estimation network MagnetO that uses activations from the pooling layer of the trained deep speaker extractor and generalization of such approach based on separate scale and offset prediction neural networks. An additional focus of this research is to estimate the impact of score normalization on the calibration performance of the system. The obtained results demonstrate that there are no serious problems if in-domain development data are used for calibration tuning. Otherwise, a trade-off between good calibration performance and threshold-free system quality arises. In most cases using adaptive s-norm helps to stabilize score distributions and to improve system performance. Meanwhile, some experiments demonstrate that novel approaches have their limits in score stabilization on several datasets.

Related papers

• Improving Predictor Reliability with Selective Recalibration [15.319277333431318]
  Recalibration is one of the most effective ways to produce reliable confidence estimates with a pre-trained model. We propose textitselective recalibration, where a selection model learns to reject some user-chosen proportion of the data. Our results show that selective recalibration consistently leads to significantly lower calibration error than a wide range of selection and recalibration baselines.
  arXiv  Detail & Related papers  (2024-10-07T18:17:31Z)
• Calibration-Aware Bayesian Learning [37.82259435084825]
  This paper proposes an integrated framework, referred to as calibration-aware Bayesian neural networks (CA-BNNs) It applies both data-dependent or data-independent regularizers while optimizing over a variational distribution as in Bayesian learning. Numerical results validate the advantages of the proposed approach in terms of expected calibration error (ECE) and reliability diagrams.
  arXiv  Detail & Related papers  (2023-05-12T14:19:15Z)
• Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise [62.997667081978825]
  In high-risk environments, deep learning models need to be able to judge their uncertainty and reject inputs when there is a significant chance of misclassification. We conduct a rigorous evaluation of the most commonly used uncertainty and robustness methods for the classification of Whole Slide Images. We observe that ensembles of methods generally lead to better uncertainty estimates as well as an increased robustness towards domain shifts and label noise.
  arXiv  Detail & Related papers  (2023-01-03T11:34:36Z)
• Large-Scale Sequential Learning for Recommender and Engineering Systems [91.3755431537592]
  In this thesis, we focus on the design of an automatic algorithms that provide personalized ranking by adapting to the current conditions. For the former, we propose novel algorithm called SAROS that take into account both kinds of feedback for learning over the sequence of interactions. The proposed idea of taking into account the neighbour lines shows statistically significant results in comparison with the initial approach for faults detection in power grid.
  arXiv  Detail & Related papers  (2022-05-13T21:09:41Z)
• Learning to Adapt to Domain Shifts with Few-shot Samples in Anomalous Sound Detection [7.631596468553607]
  Anomaly detection has many important applications, such as monitoring industrial equipment. We propose a framework that adapts to new conditions with few-shot samples. We evaluate our proposed method on a recently-released dataset of audio measurements from different machine types.
  arXiv  Detail & Related papers  (2022-04-05T00:22:25Z)
• Deep Discriminative to Kernel Density Graph for In- and Out-of-distribution Calibrated Inference [7.840433908659846]
  Deep discriminative approaches like random forests and deep neural networks have recently found applications in many important real-world scenarios. However, deploying these learning algorithms in safety-critical applications raises concerns, particularly when it comes to ensuring confidence calibration for both in-distribution and out-of-distribution data points. In this paper, we address ID and OOD calibration problems jointly.
  arXiv  Detail & Related papers  (2022-01-31T05:07:16Z)
• A Speaker Verification Backend with Robust Performance across Conditions [28.64769660252556]
  A standard method for speaker verification consists of extracting speaker embeddings with a deep neural network. This method is known to result in systems that work poorly on conditions different from those used to train the calibration model. We propose to modify the standard backend, introducing an adaptive calibrator that uses duration and other automatically extracted side-information to adapt to the conditions of the inputs.
  arXiv  Detail & Related papers  (2021-02-02T21:27:52Z)
• Capturing scattered discriminative information using a deep architecture in acoustic scene classification [49.86640645460706]
  In this study, we investigate various methods to capture discriminative information and simultaneously mitigate the overfitting problem. We adopt a max feature map method to replace conventional non-linear activations in a deep neural network. Two data augment methods and two deep architecture modules are further explored to reduce overfitting and sustain the system's discriminative power.
  arXiv  Detail & Related papers  (2020-07-09T08:32:06Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)
• Unsupervised Domain Adaptation for Acoustic Scene Classification Using Band-Wise Statistics Matching [69.24460241328521]
  Machine learning algorithms can be negatively affected by mismatches between training (source) and test (target) data distributions. We propose an unsupervised domain adaptation method that consists of aligning the first- and second-order sample statistics of each frequency band of target-domain acoustic scenes to the ones of the source-domain training dataset. We show that the proposed method outperforms the state-of-the-art unsupervised methods found in the literature in terms of both source- and target-domain classification accuracy.
  arXiv  Detail & Related papers  (2020-04-30T23:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.