Uncertainty Estimation for the Open-Set Text Classification systems

• URL: http://arxiv.org/abs/2604.08560v1
• Date: Tue, 17 Mar 2026 12:25:34 GMT
• Title: Uncertainty Estimation for the Open-Set Text Classification systems
• Abstract summary: We adapt the Holistic Uncertainty Estimation (HolUE) method for the text domain.<n>Our approach addresses two major causes of prediction errors in text recognition systems: text uncertainty that stems from ill formulated queries and gallery uncertainty that is related the ambiguity of data distribution.
• Score: 1.0557657302168184
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Accurate uncertainty estimation is essential for building robust and trustworthy recognition systems. In this paper, we consider the open-set text classification (OSTC) task - and uncertainty estimation for it. For OSTC a text sample should be classified as one of the existing classes or rejected as unknown. To account for the different uncertainty types encountered in OSTC, we adapt the Holistic Uncertainty Estimation (HolUE) method for the text domain. Our approach addresses two major causes of prediction errors in text recognition systems: text uncertainty that stems from ill formulated queries and gallery uncertainty that is related the ambiguity of data distribution. By capturing these sources, it becomes possible to predict when the system will make a recognition error. We propose a new OSTC benchmark and conduct extensive experiments on a wide range of data, utilizing the authorship attribution, intent and topic classification datasets. HolUE achieves 40-365% improvement in Prediction Rejection Ratio (PRR) over the quality-based SCF baseline across datasets: 365% on Yahoo Answers (0.79 vs 0.17 at FPIR 0.1), 347% on DBPedia (0.85 vs 0.19), 240% on PAN authorship attribution (0.51 vs 0.15 at FPIR 0.5), and 40% on CLINC150 intent classification (0.73 vs~0.52). We make public our code and protocols https://github.com/Leonid-Erlygin/text_uncertainty.git

Related papers

• $TCP_α$: Margin-Controlled Confidence estimation for reliable Music Information Retrieval [7.679660052685777]
  Deep neural networks are often overconfident, assigning high confidence even to incorrect predictions.<n>Existing targets suffer from inherent ambiguity: they assign overlapping confidence values to correct and incorrect predictions, while errors near the decision boundary receive confidence scores indistinguishable from correct predictions.<n>We propose $ TCP_$, a novel confidence target that resolves these limitations by introducing a margin-controlled penalty for misclassified samples.
  arXiv  Detail & Related papers  (2026-08-20T17:58:50Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays [0.0]
  Per-field accept/review with selective risk at most alpha is the trust contract document-extraction systems need.<n>On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes.<n>Released Apache-2.0 with seed-pinned, regression-gated procedures.
  arXiv  Detail & Related papers  (2026-07-28T17:08:50Z)
• From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings [1.8194320623567561]
  We propose an evaluation framework combining regularized regression on item text embeddings.<n>We find that item difficulty is highly predictable from text.<n>We show that embedding based regression matches leaderboard RMSE despite explaining almost no variance.
  arXiv  Detail & Related papers  (2026-07-08T08:34:46Z)
• Conformal Reliability: A New Evaluation Metric for Conditional Generation [62.761166941396844]
  Conditional generative models have recently remarkable success in various applications.<n>We propose a novel evaluation metric called reliability based on prediction set at a pre-specified confidence level.<n>We introduce Conformal ReLiability (CReL), a framework that can (i) construct a prediction set and (ii) accurately optimize the reliability score within the constructed prediction set.
  arXiv  Detail & Related papers  (2026-05-29T03:52:44Z)
• Conformal Prediction for Privacy-Preserving Machine Learning [83.88591755871734]
  Using AES-encrypted variants of the MNIST dataset, we demonstrate that Conformal Prediction methods remain effective even when applied directly in the encrypted domain.<n>Our work sets a foundation for principled uncertainty quantification in secure, privacy-aware learning systems.
  arXiv  Detail & Related papers  (2025-07-13T15:29:14Z)
• Estimating Uncertainty with Implicit Quantile Network [0.0]
  Uncertainty quantification is an important part of many performance critical applications. This paper provides a simple alternative to existing approaches such as ensemble learning and bayesian neural networks.
  arXiv  Detail & Related papers  (2024-08-26T13:33:14Z)
• Improved Out-of-Scope Intent Classification with Dual Encoding and Threshold-based Re-Classification [6.975902383951604]
  Current methodologies face difficulties with the unpredictable distribution of outliers. We present the Dual for Threshold-Based Re-Classification (DETER) to address these challenges. Our model outperforms previous benchmarks, increasing up to 13% and 5% in F1 score for known and unknown intents.
  arXiv  Detail & Related papers  (2024-05-30T11:46:42Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers [15.929238800072195]
  We present a novel study of selective prediction and the uncertainty estimation performance of 523 existing pretrained deep ImageNet classifiers. We find that distillation-based training regimes consistently yield better uncertainty estimations than other training schemes. For example, we discovered an unprecedented 99% top-1 selective accuracy on ImageNet at 47% coverage.
  arXiv  Detail & Related papers  (2023-02-23T09:25:28Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection [6.230751621285322]
  In object detection with deep neural networks, the box-wise objectness score tends to be overconfident. We present a post processing method that for any given neural network provides predictive uncertainty estimates and quality estimates.
  arXiv  Detail & Related papers  (2020-10-04T21:49:23Z)
• Uncertainty Sets for Image Classifiers using Conformal Prediction [112.54626392838163]
  We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Our method modifies an existing conformal prediction algorithm to give more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling.
  arXiv  Detail & Related papers  (2020-09-29T17:58:04Z)
• Certifying Confidence via Randomized Smoothing [151.67113334248464]
  Randomized smoothing has been shown to provide good certified-robustness guarantees for high-dimensional classification problems. Most smoothing methods do not give us any information about the confidence with which the underlying classifier makes a prediction. We propose a method to generate certified radii for the prediction confidence of the smoothed classifier.
  arXiv  Detail & Related papers  (2020-09-17T04:37:26Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.