Self-Calibrating Neural-Probabilistic Model for Authorship Verification
Under Covariate Shift
- URL: http://arxiv.org/abs/2106.11196v1
- Date: Mon, 21 Jun 2021 15:33:48 GMT
- Title: Self-Calibrating Neural-Probabilistic Model for Authorship Verification
Under Covariate Shift
- Authors: Benedikt Boenninghoff, Dorothea Kolossa, Robert M. Nickel
- Abstract summary: We are addressing two fundamental problems in authorship verification (AV)
We are expanding our framework from PAN 2020 to include Bayes factor scoring (BFS) and an uncertainty adaptation layer (UAL)
Experiments with the 2020/21 PAN AV shared task data show that the proposed method significantly reduces sensitivities to topical variations and significantly improves the system's calibration.
- Score: 14.321827655211544
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We are addressing two fundamental problems in authorship verification (AV):
Topic variability and miscalibration. Variations in the topic of two disputed
texts are a major cause of error for most AV systems. In addition, it is
observed that the underlying probability estimates produced by deep learning AV
mechanisms oftentimes do not match the actual case counts in the respective
training data. As such, probability estimates are poorly calibrated. We are
expanding our framework from PAN 2020 to include Bayes factor scoring (BFS) and
an uncertainty adaptation layer (UAL) to address both problems. Experiments
with the 2020/21 PAN AV shared task data show that the proposed method
significantly reduces sensitivities to topical variations and significantly
improves the system's calibration.
Related papers
- Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks [0.0]
It is shown that the estimation of MV, MSE and their confidence intervals becomes unreliable for heavy-tailed uncertainty and error distributions.
The same problem is expected to affect also conditional calibrations statistics, such as the popular ENCE.
arXiv Detail & Related papers (2024-02-15T16:05:35Z) - Multi-Source Domain Adaptation for Cross-Domain Fault Diagnosis of
Chemical Processes [5.119371135458389]
We provide an extensive comparison of single and multi-source unsupervised domain adaptation algorithms for Cross-Domain Fault Diagnosis (CDFD)
We show that using multiple domains during training has a positive effect, even when no adaptation is employed.
In addition, under the multiple-sources scenario, we improve classification accuracy of the no adaptation setting by 8.4% on average.
arXiv Detail & Related papers (2023-08-22T07:43:59Z) - Calibration-Aware Bayesian Learning [37.82259435084825]
This paper proposes an integrated framework, referred to as calibration-aware Bayesian neural networks (CA-BNNs)
It applies both data-dependent or data-independent regularizers while optimizing over a variational distribution as in Bayesian learning.
Numerical results validate the advantages of the proposed approach in terms of expected calibration error (ECE) and reliability diagrams.
arXiv Detail & Related papers (2023-05-12T14:19:15Z) - Bridging Precision and Confidence: A Train-Time Loss for Calibrating
Object Detection [58.789823426981044]
We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions.
Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
arXiv Detail & Related papers (2023-03-25T08:56:21Z) - Investigation of Different Calibration Methods for Deep Speaker
Embedding based Verification Systems [66.61691401921296]
This paper presents an investigation over several methods of score calibration for deep speaker embedding extractors.
An additional focus of this research is to estimate the impact of score normalization on the calibration performance of the system.
arXiv Detail & Related papers (2022-03-28T21:22:22Z) - Identifying Incorrect Classifications with Balanced Uncertainty [21.130311978327196]
Uncertainty estimation is critical for cost-sensitive deep-learning applications.
We propose the distributional imbalance to model the imbalance in uncertainty estimation as two kinds of distribution biases.
We then propose Balanced True Class Probability framework, which learns an uncertainty estimator with a novel Distributional Focal Loss objective.
arXiv Detail & Related papers (2021-10-15T11:52:31Z) - Semi-Supervised Learning with Variational Bayesian Inference and Maximum
Uncertainty Regularization [62.21716612888669]
We propose two generic methods for improving semi-supervised learning (SSL)
The first integrates weight perturbation (WP) into existing "consistency regularization" (CR) based methods.
The second method proposes a novel consistency loss called "maximum uncertainty regularization" (MUR)
arXiv Detail & Related papers (2020-12-03T09:49:35Z) - Transferable Calibration with Lower Bias and Variance in Domain
Adaptation [139.4332115349543]
Domain Adaptation (DA) enables transferring a learning machine from a labeled source domain to an unlabeled target one.
How to estimate the predictive uncertainty of DA models is vital for decision-making in safety-critical scenarios.
TransCal can be easily applied to recalibrate existing DA methods.
arXiv Detail & Related papers (2020-07-16T11:09:36Z) - Unlabelled Data Improves Bayesian Uncertainty Calibration under
Covariate Shift [100.52588638477862]
We develop an approximate Bayesian inference scheme based on posterior regularisation.
We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
arXiv Detail & Related papers (2020-06-26T13:50:19Z) - Provable tradeoffs in adversarially robust classification [96.48180210364893]
We develop and leverage new tools, including recent breakthroughs from probability theory on robust isoperimetry.
Our results reveal fundamental tradeoffs between standard and robust accuracy that grow when data is imbalanced.
arXiv Detail & Related papers (2020-06-09T09:58:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.