Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

• URL: http://arxiv.org/abs/2607.06327v2
• Date: Fri, 10 Jul 2026 15:38:02 GMT
• Title: Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
• Abstract summary: Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain.<n>We present the first large-scale evaluation of UE methods across 22 languages.
• Score: 15.831021059811045
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q\&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.

Related papers

• Choosing the Right Language Mode at Inference Time for Multilingual Reliability [3.6095388702618405]
  Large language models often struggle to reason in low- to mid-resource languages.<n>Prior work has shown that translation can improve multilingual reasoning by helping models access stronger English-centric representations.<n>This raises a central question: How much translation is needed for multilingual large language models to reason reliably.
  arXiv  Detail & Related papers  (2026-09-04T02:36:26Z)
• Shared Doubt: Zero-shot Cross-Lingual Confidence Estimation for Language Models [14.815594636224747]
  We use a lightweight linear probe that predicts answer correctness directly from intermediate representations.<n> Learned layer weights and multiple ablations reveal that confidence features concentrate in middle layers across languages.<n>While zero-shot cross-lingual performance depends on similarity to the source language, the probe provides a strong baseline without any retraining.
  arXiv  Detail & Related papers  (2026-05-29T12:25:24Z)
• Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning [58.355275813623685]
  This work looks at a critical gap in the calibration of large language models (LLMs) within multilingual settings.<n>Even in low-resource languages, model confidence can increase significantly after instruction-tuning on high-resource language SFT datasets.<n>However, improvements in accuracy are marginal or non-existent, highlighting a critical shortcoming of standard SFT for multilingual languages.
  arXiv  Detail & Related papers  (2026-01-04T04:29:12Z)
• Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models [50.34755385896279]
  Confidence calibration is crucial for the reliable deployment of Large Language Models (LLMs)<n>We conduct the first large-scale, systematic studies of multilingual calibration across six model families and over 100 languages.<n>We find that non-English languages suffer from systematically worse calibration.
  arXiv  Detail & Related papers  (2025-10-03T16:07:15Z)
• Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model [13.788758077632432]
  We introduce Pivot-Based Reinforcement Learning with Semantically Verifiable Rewards.<n>This framework enhances multilingual reasoning by circumventing the need for human-annotated data in target languages.<n>We show that our method significantly narrows the performance gap between English and other languages.
  arXiv  Detail & Related papers  (2025-09-29T22:03:11Z)
• CM-Align: Consistency-based Multilingual Alignment for Large Language Models [84.19366314925593]
  We propose a consistency-based data method to construct high-quality multilingual preference data.<n> Specifically, our method includes two parts: consistency-guided English reference selection and cross-lingual consistency-based multilingual preference data construction.
  arXiv  Detail & Related papers  (2025-09-10T12:40:49Z)
• Scaling Test-time Compute for Low-resource Languages: Multilingual Reasoning in LLMs [3.9530780161144667]
  We investigate the multilingual mechanism by which Large Language Models internally operate in a latent space biased toward their inherently dominant language.<n>We train models to generate the chain-of-thought (CoT) in English while outputting the final response in the target language, given input in the low-resource language.<n>Our experiments demonstrate that this approach, named English-Pivoted CoT Training, outperforms other baselines, with up to 28.33% improvement.
  arXiv  Detail & Related papers  (2025-04-02T16:58:36Z)
• Assessing Agentic Large Language Models in Multilingual National Bias [31.67058518564021]
  Cross-language disparities in reasoning-based recommendations remain largely unexplored.<n>This study is the first to address this gap.<n>We investigate multilingual bias in state-of-the-art LLMs by analyzing their responses to decision-making tasks across multiple languages.
  arXiv  Detail & Related papers  (2025-02-25T08:07:42Z)
• The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights [108.40766216456413]
  We propose a question alignment framework to bridge the gap between large language models' English and non-English performance. Experiment results show it can boost multilingual performance across diverse reasoning scenarios, model families, and sizes. We analyze representation space, generated response and data scales, and reveal how question translation training strengthens language alignment within LLMs.
  arXiv  Detail & Related papers  (2024-05-02T14:49:50Z)
• Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet? [82.02076369811402]
  Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning. We present an extensive and systematic comparison of the three approaches, testing them on 6 high- and low-resource languages, three different NLU tasks, and a myriad of language and domain setups. Our observations show that supervised instruction tuning has the best trade-off between performance and resource requirements.
  arXiv  Detail & Related papers  (2024-03-04T10:48:13Z)
• From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension [51.953428342923885]
  We develop a two-stage approach to enhance the model performance. The first stage targets at recall: we design a hard-learning (HL) algorithm to maximize the likelihood that the top-k predictions contain the accurate answer. The second stage focuses on precision: an answer-aware contrastive learning mechanism is developed to learn the fine difference between the accurate answer and other candidates.
  arXiv  Detail & Related papers  (2021-12-09T07:31:15Z)
• Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation [34.97086123805344]
  Recent multilingual pre-trained language models have achieved remarkable zero-shot performance. We propose a self-learning framework that further utilizes unlabeled data of target languages. We evaluate our framework with uncertainties on two cross-lingual tasks including Named Entity Recognition (NER) and Natural Language Inference (NLI) covering 40 languages in total.
  arXiv  Detail & Related papers  (2021-09-01T05:26:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.