Instance-Level Costs for Nuanced Classifier Evaluation

• URL: http://arxiv.org/abs/2605.03135v1
• Date: Mon, 04 May 2026 20:19:21 GMT
• Title: Instance-Level Costs for Nuanced Classifier Evaluation
• Abstract summary: In content moderation, medical screening, and safety-critical applications, mistakes on clear-cut cases are far more costly than errors on ambiguous ones.<n>We propose normalized excess cost (NEC), a metric that weights classification errors by per-example costs and reduces to standard error rate when costs are uniform.
• Score: 1.0481606990674208
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard classification treats all errors equally, but in content moderation, medical screening, and safety-critical applications, mistakes on clear-cut cases are far more costly than errors on ambiguous ones. We propose normalized excess cost (NEC), a metric that weights classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate -- models with 5\% error rate can achieve 1.8\% NEC -- revealing that most mistakes concentrate on ambiguous, low-cost examples. However, incorporating costs into training via loss weighting, sampling strategies, or regression yields inconsistent benefits: improvements appear only when costs are predictable from input features, as in our synthetic control, while real-world datasets show mixed or negligible gains. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs, even when cost-sensitive training offers limited benefit.

Related papers

• Cost-Aware Learning [72.31444819326795]
  We consider the problem of Cost-Aware Learning, where sampling different component functions incur different costs.<n>We propose the Cost-Aware Descent algorithm for convex functions, and derive its cost complexity to attain an error of $$.<n>We introduce Cost-Aware GRPO, an algorithm designed to reduce the cost of policy optimization while preserving performance.
  arXiv  Detail & Related papers  (2026-04-30T15:39:09Z)
• Cost-Optimal Active AI Model Evaluation [71.2069549142394]
  Development of generative AI systems requires continual evaluation, data acquisition, and annotation.<n>We develop novel, cost-aware methods for actively balancing the use of a cheap, but often inaccurate, weak rater.<n>We derive a family of cost-optimal policies for allocating a given annotation budget between weak and strong raters.
  arXiv  Detail & Related papers  (2025-06-09T17:14:41Z)
• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We present a novel variational formulation of the calibration-refinement decomposition.<n>We provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation models [3.938980910007962]
  Confidence calibration is an emerging challenge in real-world decision systems based on foundations models.<n>We propose a penalty incorporated into loss objective that penalizes incorrect classifications whenever one is made during finetuning.<n>We refer to it as textitconfidence misalignment penalty (CMP).
  arXiv  Detail & Related papers  (2025-01-29T11:54:37Z)
• Of Dice and Games: A Theory of Generalized Boosting [61.752303337418475]
  We extend the celebrated theory of boosting to incorporate both cost-sensitive and multi-objective losses.<n>We develop a comprehensive theory of cost-sensitive and multi-objective boosting, providing a taxonomy of weak learning guarantees.<n>Our characterization relies on a geometric interpretation of boosting, revealing a surprising equivalence between cost-sensitive and multi-objective losses.
  arXiv  Detail & Related papers  (2024-12-11T01:38:32Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We develop general algorithms for reducing the task of causal calibration to that of calibrating a standard (non-causal) predictive model.<n>Our results are exceedingly general, showing that essentially any existing calibration algorithm can be used in causal settings.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• On support vector machines under a multiple-cost scenario [1.743685428161914]
  Support Vector Machine (SVM) is a powerful tool in binary classification. We propose a novel SVM model in which misclassification costs are considered by incorporating performance constraints.
  arXiv  Detail & Related papers  (2023-12-22T16:12:25Z)
• Rethinking Cost-sensitive Classification in Deep Learning via Adversarial Data Augmentation [4.479834103607382]
  Cost-sensitive classification is critical in applications where misclassification errors widely vary in cost. This paper proposes a cost-sensitive adversarial data augmentation framework to make over- parameterized models cost-sensitive. Our method can effectively minimize the overall cost and reduce critical errors, while achieving comparable performance in terms of overall accuracy.
  arXiv  Detail & Related papers  (2022-08-24T19:00:30Z)
• Training Over-parameterized Models with Non-decomposable Objectives [46.62273918807789]
  We propose new cost-sensitive losses that extend the classical idea of logit adjustment to handle more general cost matrices. Our losses are calibrated, and can be further improved with distilled labels from a teacher model.
  arXiv  Detail & Related papers  (2021-07-09T19:29:33Z)
• Cost-Based Budget Active Learning for Deep Learning [0.9732863739456035]
  We propose a Cost-Based Bugdet Active Learning (CBAL) which considers the classification uncertainty as well as instance diversity in a population constrained by a budget. A principled approach based on the min-max is considered to minimize both the labeling and decision cost of the selected instances.
  arXiv  Detail & Related papers  (2020-12-09T17:42:44Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.