Well-calibrated Confidence Measures for Multi-label Text Classification with a Large Number of Labels

• URL: http://arxiv.org/abs/2312.09304v1
• Date: Thu, 14 Dec 2023 19:17:42 GMT
• Title: Well-calibrated Confidence Measures for Multi-label Text Classification with a Large Number of Labels
• Authors: Lysimachos Maltoudoglou, Andreas Paisios, Ladislav Lenc, Ji\v{r}\'i Mart\'inek, Pavel Kr\'al, Harris Papadopoulos
• Abstract summary: We present a novel approach for addressing the computational inefficiency of the Label Powerset (LP) ICP, arrising when dealing with a high number of unique labels. We apply the LP-ICP on three deep Artificial Neural Network (ANN) classifiers of two types: one based on contextualised (bert) and two on non-contextualised (word2vec) word-embeddings. Our approach deals with the increased computational burden of LP by eliminating from consideration a significant number of label-sets that will surely have p-values below the specified significance level.
• Score: 1.1833906227033337
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We extend our previous work on Inductive Conformal Prediction (ICP) for multi-label text classification and present a novel approach for addressing the computational inefficiency of the Label Powerset (LP) ICP, arrising when dealing with a high number of unique labels. We present experimental results using the original and the proposed efficient LP-ICP on two English and one Czech language data-sets. Specifically, we apply the LP-ICP on three deep Artificial Neural Network (ANN) classifiers of two types: one based on contextualised (bert) and two on non-contextualised (word2vec) word-embeddings. In the LP-ICP setting we assign nonconformity scores to label-sets from which the corresponding p-values and prediction-sets are determined. Our approach deals with the increased computational burden of LP by eliminating from consideration a significant number of label-sets that will surely have p-values below the specified significance level. This reduces dramatically the computational complexity of the approach while fully respecting the standard CP guarantees. Our experimental results show that the contextualised-based classifier surpasses the non-contextualised-based ones and obtains state-of-the-art performance for all data-sets examined. The good performance of the underlying classifiers is carried on to their ICP counterparts without any significant accuracy loss, but with the added benefits of ICP, i.e. the confidence information encapsulated in the prediction sets. We experimentally demonstrate that the resulting prediction sets can be tight enough to be practically useful even though the set of all possible label-sets contains more than $1e+16$ combinations. Additionally, the empirical error rates of the obtained prediction-sets confirm that our outputs are well-calibrated.

Related papers

• Reliable Hierarchical Operating System Fingerprinting via Conformal Prediction [62.40452053128524]
  Conformal Prediction (CP) could be wrapped around existing methods to obtain prediction sets with guaranteed coverage.<n>This work addresses these limitations by introducing and evaluating two distinct structured CP strategies.<n>While both methods satisfy validity guarantees, they expose a fundamental trade-off between level-wise efficiency and structural consistency.
  arXiv  Detail & Related papers  (2026-02-13T11:20:48Z)
• DiCaP: Distribution-Calibrated Pseudo-labeling for Semi-Supervised Multi-Label Learning [83.94574004953346]
  Semi-supervised multi-label learning aims to leverage unlabeled data to improve the model's performance.<n>Most existing methods assign equal weights to all pseudo-labels regardless of their quality.<n>We propose Distribution-Calibrated Pseudo-labeling (DiCaP), a correctness-aware framework that estimates posterior precision to calibrate pseudo-label weights.
  arXiv  Detail & Related papers  (2025-11-25T11:55:02Z)
• Semi-Supervised Conformal Prediction With Unlabeled Nonconformity Score [19.15617038007535]
  Conformal prediction (CP) is a powerful framework for uncertainty quantification.<n>In real-world applications where labeled data is often limited, standard CP can lead to coverage deviation and output overly large prediction sets.<n>We propose SemiCP, leveraging both labeled data and unlabeled data for calibration.
  arXiv  Detail & Related papers  (2025-05-27T12:57:44Z)
• Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness [53.96714099151378]
  We propose a three-step approach for parameter-efficient fine-tuning of image-text foundation models. Our method improves its two key components: minority samples identification and the robust training algorithm. Our theoretical analysis shows that our PPA enhances minority group identification and is Bayes optimal for minimizing the balanced group error.
  arXiv  Detail & Related papers  (2025-03-12T15:46:12Z)
• A Debiased Nearest Neighbors Framework for Multi-Label Text Classification [13.30576550077694]
  We introduce a DEbiased Nearest Neighbors (DENN) framework for Multi-Label Text Classification (MLTC) To address embedding alignment bias, we propose a debiased contrastive learning strategy, enhancing neighbor consistency on label co-occurrence. For confidence estimation bias, we present a debiased confidence estimation strategy, improving the adaptive combination of predictions from $k$NN and inductive binary classifications.
  arXiv  Detail & Related papers  (2024-08-06T14:00:23Z)
• Adapting Conformal Prediction to Distribution Shifts Without Labels [16.478151550456804]
  Conformal prediction (CP) enables machine learning models to output prediction sets with guaranteed coverage rate. Our goal is to improve the quality of CP-generated prediction sets using only unlabeled data from the test domain. This is achieved by two new methods called ECP and EACP, that adjust the score function in CP according to the base model's uncertainty on the unlabeled test data.
  arXiv  Detail & Related papers  (2024-06-03T15:16:02Z)
• PAC Prediction Sets Under Label Shift [52.30074177997787]
  Prediction sets capture uncertainty by predicting sets of labels rather than individual labels. We propose a novel algorithm for constructing prediction sets with PAC guarantees in the label shift setting. We evaluate our approach on five datasets.
  arXiv  Detail & Related papers  (2023-10-19T17:57:57Z)
• Ambiguity-Resistant Semi-Supervised Learning for Dense Object Detection [98.66771688028426]
  We propose a Ambiguity-Resistant Semi-supervised Learning (ARSL) for one-stage detectors. Joint-Confidence Estimation (JCE) is proposed to quantifies the classification and localization quality of pseudo labels. ARSL effectively mitigates the ambiguities and achieves state-of-the-art SSOD performance on MS COCO and PASCAL VOC.
  arXiv  Detail & Related papers  (2023-03-27T07:46:58Z)
• NP-Match: Towards a New Probabilistic Model for Semi-Supervised Learning [86.60013228560452]
  Semi-supervised learning (SSL) has been widely explored in recent years, and it is an effective way of leveraging unlabeled data. In this work, we adjust neural processes (NPs) to the semi-supervised image classification task, resulting in a new method named NP-Match. NP-Match implicitly compares data points when making predictions, and as a result, the prediction of each unlabeled data point is affected by the labeled data points.
  arXiv  Detail & Related papers  (2023-01-31T11:44:45Z)
• NP-Match: When Neural Processes meet Semi-Supervised Learning [133.009621275051]
  Semi-supervised learning (SSL) has been widely explored in recent years, and it is an effective way of leveraging unlabeled data to reduce the reliance on labeled data. In this work, we adjust neural processes (NPs) to the semi-supervised image classification task, resulting in a new method named NP-Match.
  arXiv  Detail & Related papers  (2022-07-03T15:24:31Z)
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning [53.1047775185362]
  Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process.
  arXiv  Detail & Related papers  (2021-01-15T23:29:57Z)
• Pointwise Binary Classification with Pairwise Confidence Comparisons [97.79518780631457]
  We propose pairwise comparison (Pcomp) classification, where we have only pairs of unlabeled data that we know one is more likely to be positive than the other. We link Pcomp classification to noisy-label learning to develop a progressive URE and improve it by imposing consistency regularization.
  arXiv  Detail & Related papers  (2020-10-05T09:23:58Z)
• Efficient Conformal Prediction via Cascaded Inference with Expanded Admission [43.596058175459746]
  We present a novel approach for conformal prediction (CP) We aim to identify a set of promising prediction candidates -- in place of a single prediction. This set is guaranteed to contain a correct answer with high probability.
  arXiv  Detail & Related papers  (2020-07-06T23:13:07Z)
• On Aggregation in Ensembles of Multilabel Classifiers [4.842945656927122]
  "predict then combine" (PTC) and "combine then predict" (CTP) are two principal approaches to ensemble multilabel classification. We show that PTC is the better choice for non-decomposable losses.
  arXiv  Detail & Related papers  (2020-06-21T21:43:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.