Specialists Outperform Generalists in Ensemble Classification

• URL: http://arxiv.org/abs/2107.04381v1
• Date: Fri, 9 Jul 2021 12:16:10 GMT
• Title: Specialists Outperform Generalists in Ensemble Classification
• Authors: Sascha Meyen, Frieder G\"oppert, Helen Alber, Ulrike von Luxburg, Volker H. Franz
• Abstract summary: In this paper, we address the question of whether we can determine the accuracy of the ensemble. We explicitly construct the individual classifiers that attain the upper and lower bounds: specialists and generalists.
• Score: 15.315432841707736
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Consider an ensemble of $k$ individual classifiers whose accuracies are known. Upon receiving a test point, each of the classifiers outputs a predicted label and a confidence in its prediction for this particular test point. In this paper, we address the question of whether we can determine the accuracy of the ensemble. Surprisingly, even when classifiers are combined in the statistically optimal way in this setting, the accuracy of the resulting ensemble classifier cannot be computed from the accuracies of the individual classifiers-as would be the case in the standard setting of confidence weighted majority voting. We prove tight upper and lower bounds on the ensemble accuracy. We explicitly construct the individual classifiers that attain the upper and lower bounds: specialists and generalists. Our theoretical results have very practical consequences: (1) If we use ensemble methods and have the choice to construct our individual (independent) classifiers from scratch, then we should aim for specialist classifiers rather than generalists. (2) Our bounds can be used to determine how many classifiers are at least required to achieve a desired ensemble accuracy. Finally, we improve our bounds by considering the mutual information between the true label and the individual classifier's output.

Related papers

• Mitigating Word Bias in Zero-shot Prompt-based Classifiers [55.60306377044225]
  We show that matching class priors correlates strongly with the oracle upper bound performance. We also demonstrate large consistent performance gains for prompt settings over a range of NLP tasks.
  arXiv  Detail & Related papers  (2023-09-10T10:57:41Z)
• Class-Conditional Conformal Prediction with Many Classes [60.8189977620604]
  We propose a method called clustered conformal prediction that clusters together classes having "similar" conformal scores. We find that clustered conformal typically outperforms existing methods in terms of class-conditional coverage and set size metrics.
  arXiv  Detail & Related papers  (2023-06-15T17:59:02Z)
• Anomaly Detection using Ensemble Classification and Evidence Theory [62.997667081978825]
  We present a novel approach for novel detection using ensemble classification and evidence theory. A pool selection strategy is presented to build a solid ensemble classifier. We use uncertainty for the anomaly detection approach.
  arXiv  Detail & Related papers  (2022-12-23T00:50:41Z)
• A Dataset-Level Geometric Framework for Ensemble Classifiers [0.76146285961466]
  Majority voting and weighted majority voting are two commonly used combination schemes in ensemble learning. We present a group of properties of these two combination schemes formally under a dataset-level geometric framework.
  arXiv  Detail & Related papers  (2021-06-16T09:48:12Z)
• Early Exiting with Ensemble Internal Classifiers [57.80488632985445]
  Early exiting has gained much attention in the NLP community. We propose a voting-based strategy that considers predictions of all the past internal classifiers to infer the correct label. Experimental results on various NLP tasks show that our proposed objective function and voting-based strategy can achieve better accuracy-speed trade-offs.
  arXiv  Detail & Related papers  (2021-05-28T12:54:11Z)
• Selective Classification Can Magnify Disparities Across Groups [89.14499988774985]
  We find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities. Increasing abstentions can even decrease accuracies on some groups. We train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group.
  arXiv  Detail & Related papers  (2020-10-27T08:51:30Z)
• Classifier-independent Lower-Bounds for Adversarial Robustness [13.247278149124757]
  We theoretically analyse the limits of robustness to test-time adversarial and noisy examples in classification. We use optimal transport theory to derive variational formulae for the Bayes-optimal error a classifier can make on a given classification problem. We derive explicit lower-bounds on the Bayes-optimal error in the case of the popular distance-based attacks.
  arXiv  Detail & Related papers  (2020-06-17T16:46:39Z)
• Quantifying the Uncertainty of Precision Estimates for Rule based Text Classifiers [0.0]
  Rule based classifiers that use the presence and absence of key sub-strings to make classification decisions have a natural mechanism for quantifying the uncertainty of their precision. For a binary classifier, the key insight is to treat partitions of the sub-string set induced by the documents as Bernoulli random variables. The utility of this approach is demonstrated with a benchmark problem.
  arXiv  Detail & Related papers  (2020-05-19T03:51:47Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.