Towards More Fine-grained and Reliable NLP Performance Prediction
- URL: http://arxiv.org/abs/2102.05486v1
- Date: Wed, 10 Feb 2021 15:23:20 GMT
- Title: Towards More Fine-grained and Reliable NLP Performance Prediction
- Authors: Zihuiwen Ye, Pengfei Liu, Jinlan Fu, Graham Neubig
- Abstract summary: We make two contributions to improving performance prediction for NLP tasks.
First, we examine performance predictors for holistic measures of accuracy like F1 or BLEU.
Second, we propose methods to understand the reliability of a performance prediction model from two angles: confidence intervals and calibration.
- Score: 85.78131503006193
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Performance prediction, the task of estimating a system's performance without
performing experiments, allows us to reduce the experimental burden caused by
the combinatorial explosion of different datasets, languages, tasks, and
models. In this paper, we make two contributions to improving performance
prediction for NLP tasks. First, we examine performance predictors not only for
holistic measures of accuracy like F1 or BLEU but also fine-grained performance
measures such as accuracy over individual classes of examples. Second, we
propose methods to understand the reliability of a performance prediction model
from two angles: confidence intervals and calibration. We perform an analysis
of four types of NLP tasks, and both demonstrate the feasibility of
fine-grained performance prediction and the necessity to perform reliability
analysis for performance prediction methods in the future. We make our code
publicly available: \url{https://github.com/neulab/Reliable-NLPPP}
Related papers
- Uncertainty-Aware Performance Prediction for Highly Configurable
Software Systems via Bayesian Neural Networks [12.607426130997336]
We propose a Bayesian deep learning based method, namely BDLPerf, that can incorporate uncertainty into the prediction model.
We develop a novel uncertainty calibration technique to ensure the reliability of the confidence intervals generated by a Bayesian prediction model.
Our experimental results on 10 real-world systems show that BDLPerf achieves higher accuracy than existing approaches.
arXiv Detail & Related papers (2022-12-27T04:39:26Z) - Performance Prediction Under Dataset Shift [1.1602089225841632]
We study the generalization capabilities of various performance prediction models to new domains by learning on generated synthetic perturbations.
We propose a natural and effortless uncertainty estimation of the predicted accuracy that ensures reliable use of performance predictors.
arXiv Detail & Related papers (2022-06-21T19:40:58Z) - Pathologies of Pre-trained Language Models in Few-shot Fine-tuning [50.3686606679048]
We show that pre-trained language models with few examples show strong prediction bias across labels.
Although few-shot fine-tuning can mitigate the prediction bias, our analysis shows models gain performance improvement by capturing non-task-related features.
These observations alert that pursuing model performance with fewer examples may incur pathological prediction behavior.
arXiv Detail & Related papers (2022-04-17T15:55:18Z) - Non-Clairvoyant Scheduling with Predictions Revisited [77.86290991564829]
In non-clairvoyant scheduling, the task is to find an online strategy for scheduling jobs with a priori unknown processing requirements.
We revisit this well-studied problem in a recently popular learning-augmented setting that integrates (untrusted) predictions in algorithm design.
We show that these predictions have desired properties, admit a natural error measure as well as algorithms with strong performance guarantees.
arXiv Detail & Related papers (2022-02-21T13:18:11Z) - Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions.
In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data.
We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
arXiv Detail & Related papers (2022-01-11T23:01:12Z) - Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic
Regression [51.770998056563094]
Probabilistic Gradient Boosting Machines (PGBM) is a method to create probabilistic predictions with a single ensemble of decision trees.
We empirically demonstrate the advantages of PGBM compared to existing state-of-the-art methods.
arXiv Detail & Related papers (2021-06-03T08:32:13Z) - Learning Prediction Intervals for Model Performance [1.433758865948252]
We propose a method to compute prediction intervals for model performance.
We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines.
arXiv Detail & Related papers (2020-12-15T21:32:03Z) - Towards Improving Selective Prediction Ability of NLP Systems [24.774450633678125]
We propose a method that improves probability estimates of models by calibrating them using prediction confidence and difficulty score of instances.
We instantiate our method with Natural Language Inference (NLI) and Duplicate Detection (DD) tasks and evaluate it in both In-Domain (IID) and Out-of-Domain (OOD) settings.
arXiv Detail & Related papers (2020-08-21T08:46:36Z) - Robust Validation: Confident Predictions Even When Distributions Shift [19.327409270934474]
We describe procedures for robust predictive inference, where a model provides uncertainty estimates on its predictions rather than point predictions.
We present a method that produces prediction sets (almost exactly) giving the right coverage level for any test distribution in an $f$-divergence ball around the training population.
An essential component of our methodology is to estimate the amount of expected future data shift and build robustness to it.
arXiv Detail & Related papers (2020-08-10T17:09:16Z) - Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples.
We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries.
We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
arXiv Detail & Related papers (2020-02-27T10:22:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.