Selective Classification Under Distribution Shifts

• URL: http://arxiv.org/abs/2405.05160v1
• Date: Wed, 8 May 2024 15:52:50 GMT
• Title: Selective Classification Under Distribution Shifts
• Authors: Hengyue Liang, Le Peng, Ju Sun,
• Abstract summary: In selective classification, a classifier abstains from making predictions that are likely to be wrong to avoid excessive errors. We propose an SC framework that takes into account distribution shifts. We show that our proposed score functions are more effective and reliable than the existing ones for generalized SC.
• Score: 2.6541808384534478
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In selective classification (SC), a classifier abstains from making predictions that are likely to be wrong to avoid excessive errors. To deploy imperfect classifiers -- imperfect either due to intrinsic statistical noise of data or for robustness issue of the classifier or beyond -- in high-stakes scenarios, SC appears to be an attractive and necessary path to follow. Despite decades of research in SC, most previous SC methods still focus on the ideal statistical setting only, i.e., the data distribution at deployment is the same as that of training, although practical data can come from the wild. To bridge this gap, in this paper, we propose an SC framework that takes into account distribution shifts, termed generalized selective classification, that covers label-shifted (or out-of-distribution) and covariate-shifted samples, in addition to typical in-distribution samples, the first of its kind in the SC literature. We focus on non-training-based confidence-score functions for generalized SC on deep learning (DL) classifiers and propose two novel margin-based score functions. Through extensive analysis and experiments, we show that our proposed score functions are more effective and reliable than the existing ones for generalized SC on a variety of classification tasks and DL classifiers.

Related papers

• Liberating Seen Classes: Boosting Few-Shot and Zero-Shot Text Classification via Anchor Generation and Classification Reframing [38.84431954053434]
  Few-shot and zero-shot text classification aim to recognize samples from novel classes with limited labeled samples or no labeled samples at all. We propose a simple and effective strategy for few-shot and zero-shot text classification.
  arXiv  Detail & Related papers  (2024-05-06T15:38:32Z)
• Proposal Distribution Calibration for Few-Shot Object Detection [65.19808035019031]
  In few-shot object detection (FSOD), the two-step training paradigm is widely adopted to mitigate the severe sample imbalance. Unfortunately, the extreme data scarcity aggravates the proposal distribution bias, hindering the RoI head from evolving toward novel classes. We introduce a simple yet effective proposal distribution calibration (PDC) approach to neatly enhance the localization and classification abilities of the RoI head.
  arXiv  Detail & Related papers  (2022-12-15T05:09:11Z)
• Parametric Classification for Generalized Category Discovery: A Baseline Study [70.73212959385387]
  Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples. We investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem. We propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers.
  arXiv  Detail & Related papers  (2022-11-21T18:47:11Z)
• One-Class Risk Estimation for One-Class Hyperspectral Image Classification [8.206701378422968]
  Hyperspectral imagery (HSI) one-class classification is aimed at identifying a single target class from the HSI. Deep learning-based methods are currently the mainstream to overcome distribution overlap in HSI multiclassification. In this article, a weakly supervised deep HSI one-class classification, HOneCls, is proposed.
  arXiv  Detail & Related papers  (2022-10-27T14:15:13Z)
• AUC-based Selective Classification [5.406386303264086]
  We propose a model-agnostic approach to associate a selection function to a given binary classifier. We provide both theoretical justifications and a novel algorithm, called $AUCross$, to achieve such a goal. Experiments show that $AUCross$ succeeds in trading-off coverage for AUC, improving over existing selective classification methods targeted at optimizing accuracy.
  arXiv  Detail & Related papers  (2022-10-19T16:29:50Z)
• Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
  We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
  arXiv  Detail & Related papers  (2022-01-26T19:59:14Z)
• When in Doubt: Improving Classification Performance with Alternating Normalization [57.39356691967766]
  We introduce Classification with Alternating Normalization (CAN), a non-parametric post-processing step for classification. CAN improves classification accuracy for challenging examples by re-adjusting their predicted class probability distribution. We empirically demonstrate its effectiveness across a diverse set of classification tasks.
  arXiv  Detail & Related papers  (2021-09-28T02:55:42Z)
• Semi-Supervised Learning of Classifiers from a Statistical Perspective: A Brief Review [1.6752182911522517]
  We provide here a review of statistical SSL approaches to forming a classifier. We focus on the recent result that a classifier formed from a partially classified sample can actually have smaller expected error rate than that if the sample were completely classified.
  arXiv  Detail & Related papers  (2021-04-08T20:41:57Z)
• Selective Classification via One-Sided Prediction [54.05407231648068]
  One-sided prediction (OSP) based relaxation yields an SC scheme that attains near-optimal coverage in the practically relevant high target accuracy regime. We theoretically derive bounds generalization for SC and OSP, and empirically we show that our scheme strongly outperforms state of the art methods in coverage at small error levels.
  arXiv  Detail & Related papers  (2020-10-15T16:14:27Z)
• Solving Long-tailed Recognition with Deep Realistic Taxonomic Classifier [68.38233199030908]
  Long-tail recognition tackles the natural non-uniformly distributed data in realworld scenarios. While moderns perform well on populated classes, its performance degrades significantly on tail classes. Deep-RTC is proposed as a new solution to the long-tail problem, combining realism with hierarchical predictions.
  arXiv  Detail & Related papers  (2020-07-20T05:57:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.