Better Multi-class Probability Estimates for Small Data Sets

• URL: http://arxiv.org/abs/2001.11242v1
• Date: Thu, 30 Jan 2020 10:21:26 GMT
• Title: Better Multi-class Probability Estimates for Small Data Sets
• Authors: Tuomo Alasalmi, Jaakko Suutala, Heli Koskim\"aki and Juha R\"oning
• Abstract summary: We show that Data Generation and Grouping algorithm can be used to solve multi-class problems. Our experiments show that calibration error can be decreased using the proposed approach and the additional computational cost is acceptable.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many classification applications require accurate probability estimates in addition to good class separation but often classifiers are designed focusing only on the latter. Calibration is the process of improving probability estimates by post-processing but commonly used calibration algorithms work poorly on small data sets and assume the classification task to be binary. Both of these restrictions limit their real-world applicability. Previously introduced Data Generation and Grouping algorithm alleviates the problem posed by small data sets and in this article, we will demonstrate that its application to multi-class problems is also possible which solves the other limitation. Our experiments show that calibration error can be decreased using the proposed approach and the additional computational cost is acceptable.

Related papers

• Confidence Calibration of Classifiers with Many Classes [5.018156030818883]
  For classification models based on neural networks, the maximum predicted class probability is often used as a confidence score. This score rarely predicts well the probability of making a correct prediction and requires a post-processing calibration step.
  arXiv  Detail & Related papers  (2024-11-05T10:51:01Z)
• Class-wise and reduced calibration methods [0.0]
  We show how a reduced calibration method transforms the original problem into a simpler one. Second, we propose class-wise calibration methods, based on building on a phenomenon called neural collapse. Applying the two methods together results in class-wise reduced calibration algorithms, which are powerful tools for reducing the prediction and per-class calibration errors.
  arXiv  Detail & Related papers  (2022-10-07T17:13:17Z)
• An Embarrassingly Simple Approach to Semi-Supervised Few-Shot Learning [58.59343434538218]
  We propose a simple but quite effective approach to predict accurate negative pseudo-labels of unlabeled data from an indirect learning perspective. Our approach can be implemented in just few lines of code by only using off-the-shelf operations.
  arXiv  Detail & Related papers  (2022-09-28T02:11:34Z)
• Continual Learning For On-Device Environmental Sound Classification [63.81276321857279]
  We propose a simple and efficient continual learning method for on-device environmental sound classification. Our method selects the historical data for the training by measuring the per-sample classification uncertainty.
  arXiv  Detail & Related papers  (2022-07-15T12:13:04Z)
• Learning from Multiple Unlabeled Datasets with Partial Risk Regularization [80.54710259664698]
  In this paper, we aim to learn an accurate classifier without any class labels. We first derive an unbiased estimator of the classification risk that can be estimated from the given unlabeled sets. We then find that the classifier obtained as such tends to cause overfitting as its empirical risks go negative during training. Experiments demonstrate that our method effectively mitigates overfitting and outperforms state-of-the-art methods for learning from multiple unlabeled sets.
  arXiv  Detail & Related papers  (2022-07-04T16:22:44Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View [82.80085730891126]
  We provide the first modernally precise analysis of linear multiclass classification. Our analysis reveals that the classification accuracy is highly distribution-dependent. The insights gained may pave the way for a precise understanding of other classification algorithms.
  arXiv  Detail & Related papers  (2020-11-16T05:17:29Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)
• Probabilistic Diagnostic Tests for Degradation Problems in Supervised Learning [0.0]
  Problems such as class imbalance, overlapping, small-disjuncts, noisy labels, and sparseness limit accuracy in classification algorithms. Probability diagnostic model based on identifying signs and symptoms of each problem is presented. Behavior and performance of several supervised algorithms are studied when training sets have such problems.
  arXiv  Detail & Related papers  (2020-04-06T20:32:35Z)
• Better Classifier Calibration for Small Data Sets [0.0]
  We show how generating more data for calibration is able to improve calibration algorithm performance. The proposed approach adds computational cost but considering that the main use case is with small data sets this extra computational cost stays insignificant.
  arXiv  Detail & Related papers  (2020-02-24T12:27:21Z)
• Temporal Probability Calibration [0.0]
  We consider calibrating models that produce class probability estimates from sequences of data, focusing on the case where predictions are obtained from incomplete sequences. We show that traditional calibration techniques are not sufficiently expressive for this task, and propose methods that adapt calibration schemes depending on the length of an input sequence.
  arXiv  Detail & Related papers  (2020-02-07T06:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.