Optimally Efficient Sequential Calibration of Binary Classifiers to Minimize Classification Error

• URL: http://arxiv.org/abs/2108.08780v1
• Date: Thu, 19 Aug 2021 16:42:40 GMT
• Title: Optimally Efficient Sequential Calibration of Binary Classifiers to Minimize Classification Error
• Abstract summary: We show that for the given target variables and the score outputs of an estimator, an 'optimal' soft mapping is a hard mapping that maps the score values to probabilities. We propose a sequential merger approach, which produces an 'optimal' hard mapping sequentially with each incoming new sample.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we aim to calibrate the score outputs of an estimator for the binary classification problem by finding an 'optimal' mapping to class probabilities, where the 'optimal' mapping is in the sense that minimizes the classification error (or equivalently, maximizes the accuracy). We show that for the given target variables and the score outputs of an estimator, an 'optimal' soft mapping, which monotonically maps the score values to probabilities, is a hard mapping that maps the score values to $0$ and $1$. We show that for class weighted (where the accuracy for one class is more important) and sample weighted (where the samples' accurate classifications are not equally important) errors, or even general linear losses; this hard mapping characteristic is preserved. We propose a sequential recursive merger approach, which produces an 'optimal' hard mapping (for the observed samples so far) sequentially with each incoming new sample. Our approach has a logarithmic in sample size time complexity, which is optimally efficient.

Related papers

• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Adaptive Sampled Softmax with Inverted Multi-Index: Methods, Theory and Applications [79.53938312089308]
  The MIDX-Sampler is a novel adaptive sampling strategy based on an inverted multi-index approach. Our method is backed by rigorous theoretical analysis, addressing key concerns such as sampling bias, gradient bias, convergence rates, and generalization error bounds.
  arXiv  Detail & Related papers  (2025-01-15T04:09:21Z)
• Fractional Naive Bayes (FNB): non-convex optimization for a parsimonious weighted selective naive Bayes classifier [0.0]
  We supervised classification for datasets with a very large number of input variables. We propose a regularization of the model log-like Baylihood. The various proposed algorithms result in optimization-based weighted na"ivees scheme.
  arXiv  Detail & Related papers  (2024-09-17T11:54:14Z)
• Intra-class Adaptive Augmentation with Neighbor Correction for Deep Metric Learning [99.14132861655223]
  We propose a novel intra-class adaptive augmentation (IAA) framework for deep metric learning. We reasonably estimate intra-class variations for every class and generate adaptive synthetic samples to support hard samples mining. Our method significantly improves and outperforms the state-of-the-art methods on retrieval performances by 3%-6%.
  arXiv  Detail & Related papers  (2022-11-29T14:52:38Z)
• Fair and Optimal Classification via Post-Processing [10.163721748735801]
  This paper provides a complete characterization of the inherent tradeoff of demographic parity on classification problems. We show that the minimum error rate achievable by randomized and attribute-aware fair classifiers is given by the optimal value of a Wasserstein-barycenter problem.
  arXiv  Detail & Related papers  (2022-11-03T00:04:04Z)
• EXACT: How to Train Your Accuracy [6.144680854063938]
  We propose a new optimization framework by introducing ascentity to a model's output and optimizing expected accuracy. Experiments on linear models and deep image classification show that the proposed optimization method is a powerful alternative to widely used classification losses.
  arXiv  Detail & Related papers  (2022-05-19T15:13:00Z)
• Efficient and Differentiable Conformal Prediction with General Function Classes [96.74055810115456]
  We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
  arXiv  Detail & Related papers  (2022-02-22T18:37:23Z)
• Model-Change Active Learning in Graph-Based Semi-Supervised Learning [5.174023161939957]
  "Model Change" active learning quantifies the resulting change by introducing the additional label(s) We consider a family of convex loss functions for which the acquisition function can be efficiently approximated using the Laplace approximation of the posterior distribution.
  arXiv  Detail & Related papers  (2021-10-14T21:47:10Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Improving Calibration for Long-Tailed Recognition [68.32848696795519]
  We propose two methods to improve calibration and performance in such scenarios. For dataset bias due to different samplers, we propose shifted batch normalization. Our proposed methods set new records on multiple popular long-tailed recognition benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-01T13:55:21Z)
• Making Affine Correspondences Work in Camera Geometry Computation [62.7633180470428]
  Local features provide region-to-region rather than point-to-point correspondences. We propose guidelines for effective use of region-to-region matches in the course of a full model estimation pipeline. Experiments show that affine solvers can achieve accuracy comparable to point-based solvers at faster run-times.
  arXiv  Detail & Related papers  (2020-07-20T12:07:48Z)
• Ranking a set of objects: a graph based least-square approach [70.7866286425868]
  We consider the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of equal workers. We propose a class of non-adaptive ranking algorithms that rely on a least-squares intrinsic optimization criterion for the estimation of qualities.
  arXiv  Detail & Related papers  (2020-02-26T16:19:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.