Improving Selective Classification with Pairwise Queries for Binary Classification

• URL: http://arxiv.org/abs/2605.30615v1
• Date: Thu, 28 May 2026 22:08:08 GMT
• Title: Improving Selective Classification with Pairwise Queries for Binary Classification
• Abstract summary: In selective classification, a model predicts labels of data samples where it is confident, and abstains from predicting labels for samples on which it is not confident.<n>The estimate of a model's confidence might be inconsistent with the model's predictions, which can lead to high error on non-rejected points.<n>These pairwise queries can detect high-error samples and be incorporated into selective classification techniques to reduce the error on non-rejected samples.
• Score: 18.84830201314633
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In selective classification, a model predicts the labels of data samples where it is confident, and abstains from predicting labels for samples on which it is not confident. The rejected samples are often labeled by an expert, which is expensive. The budget for the expert is best utilized when the model has low error on non-rejected samples. However, the estimate of a model's confidence might be inconsistent with the model's predictions, which can lead to high error on non-rejected points. Such situations can readily occur in in-context binary classification by LLMs. To remedy this, we propose making additional pairwise queries to the same model. These pairwise queries can detect high-error samples and be incorporated into selective classification techniques to reduce the error on non-rejected samples. Theoretically, we establish the conditions under which a simple algorithm using pairwise queries outperforms an inconsistent confidence estimate. We support this insight through extensive experiments for $1$ synthetic and $4$ in-context learning-based real binary classification datasets. In all these cases, we show that our algorithms, using pairwise queries, obtain a better accuracy-cost tradeoff than using only the raw confidence estimates, for instance, the LLM's next-token logits.

Related papers

• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled yet Hard-to-Learn Samples in Noisy Data [18.111971239860836]
  We propose a novel sample selection method for image classification in the presence of noisy labels.<n>Our goal is to accurately distinguish correctly labeled yet hard-to-learn samples from mislabeled ones.<n>Our method functions as a plug-and-play component that can be seamlessly integrated into existing sample selection techniques.
  arXiv  Detail & Related papers  (2025-04-24T12:07:14Z)
• A3Rank: Augmentation Alignment Analysis for Prioritizing Overconfident Failing Samples for Deep Learning Models [2.6499018693213316]
  We propose a novel test case prioritization technique with augmentation alignment analysis. $A3$Rank can effectively rank failing samples escaping from the checking of confidence-based rejectors. We also provide a framework to construct a detector devoted to augmenting these rejectors to defend these failing samples.
  arXiv  Detail & Related papers  (2024-07-19T08:32:10Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Shrinking Class Space for Enhanced Certainty in Semi-Supervised Learning [59.44422468242455]
  We propose a novel method dubbed ShrinkMatch to learn uncertain samples. For each uncertain sample, it adaptively seeks a shrunk class space, which merely contains the original top-1 class. We then impose a consistency regularization between a pair of strongly and weakly augmented samples in the shrunk space to strive for discriminative representations.
  arXiv  Detail & Related papers  (2023-08-13T14:05:24Z)
• Improving group robustness under noisy labels using predictive uncertainty [0.9449650062296823]
  We use the predictive uncertainty of a model to improve the worst-group accuracy under noisy labels. We propose a novel ENtropy based Debiasing (END) framework that prevents models from learning the spurious cues while being robust to the noisy labels.
  arXiv  Detail & Related papers  (2022-12-14T04:40:50Z)
• Binary classification with ambiguous training data [69.50862982117127]
  In supervised learning, we often face with ambiguous (A) samples that are difficult to label even by domain experts. This problem is substantially different from semi-supervised learning since unlabeled samples are not necessarily difficult samples.
  arXiv  Detail & Related papers  (2020-11-05T00:53:58Z)
• Identifying Wrongly Predicted Samples: A Method for Active Learning [6.976600214375139]
  We propose a simple sample selection criterion that moves beyond uncertainty. We show state-of-the-art results and better rates at identifying wrongly predicted samples.
  arXiv  Detail & Related papers  (2020-10-14T09:00:42Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Harnessing Adversarial Distances to Discover High-Confidence Errors [0.0]
  We investigate the problem of finding errors at rates greater than expected given model confidence. We propose a query-efficient and novel search technique that is guided by adversarial perturbations.
  arXiv  Detail & Related papers  (2020-06-29T13:44:16Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.