To Predict or Not to Predict? Towards reliable uncertainty estimation in the presence of noise

• URL: http://arxiv.org/abs/2603.07330v1
• Date: Sat, 07 Mar 2026 20:22:20 GMT
• Title: To Predict or Not to Predict? Towards reliable uncertainty estimation in the presence of noise
• Abstract summary: This study examines the role of uncertainty estimation (UE) methods in multilingual text classification under noisy and non-topical conditions.<n>We evaluate a range of UE techniques against a range of metrics to assess their contribution to making more robust predictions.
• Score: 1.9746060146273674
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This study examines the role of uncertainty estimation (UE) methods in multilingual text classification under noisy and non-topical conditions. Using a complex-vs-simple sentence classification task across several languages, we evaluate a range of UE techniques against a range of metrics to assess their contribution to making more robust predictions. Results indicate that while methods relying on softmax outputs remain competitive in high-resource in-domain settings, their reliability declines in low-resource or domain-shift scenarios. In contrast, Monte Carlo dropout approaches demonstrate consistently strong performance across all languages, offering more robust calibration, stable decision thresholds, and greater discriminative power even under adverse conditions. We further demonstrate the positive impact of UE on non-topical classification: abstaining from predicting the 10\% most uncertain instances increases the macro F1 score from 0.81 to 0.85 in the Readme task. By integrating UE with trustworthiness metrics, this study provides actionable insights for developing more reliable NLP systems in real-world multilingual environments. See https://github.com/Nouran-Khallaf/To-Predict-or-Not-to-Predict

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs [15.831021059811045]
  Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain.<n>We present the first large-scale evaluation of UE methods across 22 languages.
  arXiv  Detail & Related papers  (2026-07-07T14:25:09Z)
• Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferral [0.0]
  This article presents a reliability-oriented framework for classifying free-text orthopedic notes in English, Hindi and Punjabi.<n>We compare task-aligned multilingual transformer encoders, a task-fine-tuned DistilBERT baseline, zero-shot instruction-tuned large language models (LLMs) and a domain-adaptive encoder, IndicBERT-HPA.
  arXiv  Detail & Related papers  (2026-05-29T16:30:45Z)
• Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models [54.041320081289996]
  We present the first systematic empirical study of uncertainty estimation in audio-aware large language models (ALLMs)<n>We benchmark five representative methods, including predictive entropy, length-normalized entropy, semantic entropy, discrete semantic entropy, and P(True)<n>Our results reveal two key findings. First, semantic-level and verification-based methods consistently outperform token-level baselines on general audio reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-04-28T12:56:22Z)
• Towards Reliable Truth-Aligned Uncertainty Estimation in Large Language Models [54.51264434040939]
  Uncertainty estimation (UE) aims to detect hallucinated outputs of large language models (LLMs)<n>We formalise this phenomenon as proxy failure, since most UE metrics originate from model behaviour.<n>We propose Truth AnChoring (TAC), a post-hoc calibration method to remedy UE metrics.
  arXiv  Detail & Related papers  (2026-04-01T03:42:16Z)
• Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning [58.355275813623685]
  This work looks at a critical gap in the calibration of large language models (LLMs) within multilingual settings.<n>Even in low-resource languages, model confidence can increase significantly after instruction-tuning on high-resource language SFT datasets.<n>However, improvements in accuracy are marginal or non-existent, highlighting a critical shortcoming of standard SFT for multilingual languages.
  arXiv  Detail & Related papers  (2026-01-04T04:29:12Z)
• Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation [20.726685669562496]
  Hallucinations are a common issue that undermine the reliability of large language models (LLMs)<n>Recent studies have identified a subset of hallucinations, known as confabulations, which arise due to predictive uncertainty of LLMs.<n>To detect confabulations, various methods for estimating predictive uncertainty in natural language generation (NLG) have been developed.
  arXiv  Detail & Related papers  (2025-10-02T17:54:09Z)
• Can Large Language Models Express Uncertainty Like Human? [71.27418419522884]
  We release the first diverse, large-scale dataset of hedging expressions with human-annotated confidence scores.<n>We conduct the first systematic study of linguistic confidence across modern large language models.
  arXiv  Detail & Related papers  (2025-09-29T02:34:30Z)
• Why Uncertainty Estimation Methods Fall Short in RAG: An Axiomatic Analysis [15.553942864736989]
  Uncertainty Estimation (UE) quantifies the model's confidence and helps users assess response reliability.<n>This paper shows that current UE methods cannot reliably assess correctness in the Retrieval-Augmented Generation setting.<n>We propose an axiomatic framework to identify deficiencies in existing methods and guide the development of improved approaches.
  arXiv  Detail & Related papers  (2025-05-12T11:47:42Z)
• Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
  We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection.<n>We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
  arXiv  Detail & Related papers  (2024-09-23T07:55:35Z)
• Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks [26.336947440529713]
  Trustworthiness in model predictions is crucial for safety-critical applications in the real world.<n>Deep neural networks often suffer from the issues of uncertainty estimation, such as miscalibration.<n>We propose $k$-Nearest Neighbor Uncertainty Estimation ($k$NN-UE), which uses not only the distances from the neighbors, but also the ratio of labels in the neighbors.
  arXiv  Detail & Related papers  (2024-07-02T10:33:31Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)
• MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models [36.33453112932689]
  This paper introduces a comprehensive investigation of Multilingual Confidence estimation (MlingConf) on Large Language Models (LLMs)<n>The benchmark comprises four meticulously checked and human-evaluated high-quality multilingual datasets for LA tasks and one for the LS task tailored to specific social, cultural, and geographical contexts of a language.<n>Experiments reveal that on LA tasks English exhibits notable linguistic dominance in confidence estimations than other languages, while on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance in multilingual confidence estimations.
  arXiv  Detail & Related papers  (2024-02-21T08:20:06Z)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [69.83976050879318]
  In large language models (LLMs), identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions.
  arXiv  Detail & Related papers  (2023-11-15T05:58:35Z)
• Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation [34.97086123805344]
  Recent multilingual pre-trained language models have achieved remarkable zero-shot performance. We propose a self-learning framework that further utilizes unlabeled data of target languages. We evaluate our framework with uncertainties on two cross-lingual tasks including Named Entity Recognition (NER) and Natural Language Inference (NLI) covering 40 languages in total.
  arXiv  Detail & Related papers  (2021-09-01T05:26:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.