Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs

• URL: http://arxiv.org/abs/2605.07806v1
• Date: Fri, 08 May 2026 14:41:35 GMT
• Title: Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
• Abstract summary: We propose a multidimensional perspective on model self-assessment.<n>We elicit six appraisal-based dimensions of self-assessment, alongside confidence.<n>We find that competence-related appraisal dimensions, particularly effort and ability, consistently match or outperform confidence.
• Score: 3.532798393283516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly used in settings where reliable self-assessment is critical. Assessing model reliability has evolved from using probabilistic correctness estimates to, more recently, eliciting verbalized confidence. Confidence, however, has been shown to be an inconsistent and overoptimistic predictor of model correctness. Drawing on cognitive appraisal theory, a framework from human psychology that decomposes self-evaluation into multiple components, we propose a multidimensional perspective on model self-assessment. We elicit six appraisal-based dimensions of self-assessment, alongside confidence, and evaluate their utility for predicting model failure across 12 LLMs and 38 tasks spanning eight domains. We find that competence-related appraisal dimensions, particularly effort and ability, consistently match or outperform confidence across most settings. Effort additionally yields less overoptimistic estimates that remain stable across model sizes. In contrast, affective dimensions provide marginally predictive signals. Furthermore, the most informative dimension varies systematically with task characteristics: effort is most predictive for reasoning-intensive tasks, while ability and confidence dominate on retrieval-oriented tasks. Broadly, our findings indicate that structured multidimensional self-assessment is a promising approach to improving the reliability and safety of language model deployment across diverse real-world settings.

Related papers

• Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed [53.558457942993186]
  Small Language Models offer promising potential in resource-constrained scenarios.<n>Existing approaches to building SLMs typically follow two paths: training compact models from scratch, or compressing larger pre-trained models.<n>We present a comprehensive evaluation of SLM trustworthiness across multiple dimensions, including fairness, robustness, privacy, and ethics.
  arXiv  Detail & Related papers  (2026-08-12T12:14:02Z)
• Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs [60.80435138270317]
  We apply reinforcement learning with metacognitive feedback to the problem of faithful calibration.<n>Experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks.<n> RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits.
  arXiv  Detail & Related papers  (2026-06-30T17:56:01Z)
• Linking Perception, Confidence and Accuracy in MLLMs [15.540652692247098]
  Confidence-Driven Reinforcement Learning (CDRL) uses original-noise image pairs to enhance sensitivity and robustly calibrate the model's confidence.<n> Confidence-Aware Test-Time Scaling (CA-TTS) dynamically coordinates Self-Consistency, Self-Reflection, and Visual Self-Check modules guided by confidence signals.<n>Our integrated framework establishes new state-of-the-art results with consistent 8.8% gains across four benchmarks.
  arXiv  Detail & Related papers  (2026-03-12T16:47:42Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• Confidence Estimation for LLMs in Multi-turn Interactions [48.081802290688394]
  This work presents the first systematic study of confidence estimation in multi-turn interactions.<n>We establish a formal evaluation framework grounded in two key desideratas: per-turn calibration and monotonicity of confidence.<n>Our work provides a foundational methodology for developing more reliable and trustworthy conversational agents.
  arXiv  Detail & Related papers  (2026-01-05T14:58:04Z)
• BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents [58.05949210993854]
  We investigate whether search agents have the ability to communicate their own confidence through verbalized confidence scores after long sequences of actions.<n>We propose Test-Time Scaling (TTS) methods that use confidence scores to determine answer quality, encourage the model to try again until reaching a satisfactory confidence level.
  arXiv  Detail & Related papers  (2025-10-27T15:58:51Z)
• Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models [15.158475816860427]
  Uncertainty is essential for assessing the reliability and trustworthiness of modern AI systems.<n> verbalized uncertainty, where models express their confidence through natural language, has emerged as a lightweight and interpretable solution.<n>However, its effectiveness in vision-language models (VLMs) remains insufficiently studied.
  arXiv  Detail & Related papers  (2025-05-26T17:16:36Z)
• On the Fairness, Diversity and Reliability of Text-to-Image Generative Models [68.62012304574012]
  multimodal generative models have sparked critical discussions on their reliability, fairness and potential for misuse.<n>We propose an evaluation framework to assess model reliability by analyzing responses to global and local perturbations in the embedding space.<n>Our method lays the groundwork for detecting unreliable, bias-injected models and tracing the provenance of embedded biases.
  arXiv  Detail & Related papers  (2024-11-21T09:46:55Z)
• Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models [14.5291643644017]
  We introduce the concept of Confidence-Probability Alignment. We probe the alignment between models' internal and expressed confidence. Among the models analyzed, OpenAI's GPT-4 showed the strongest confidence-probability alignment.
  arXiv  Detail & Related papers  (2024-05-25T15:42:04Z)
• Multi-Perspective Consistency Enhances Confidence Estimation in Large Language Models [27.63938857490995]
  This work focuses on improving the confidence estimation of large language models. Considering the fragility of self-awareness in language models, we introduce a Multi-Perspective Consistency (MPC) method. The experimental results on eight publicly available datasets show that our MPC achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-02-17T13:37:39Z)
• Improving the Reliability for Confidence Estimation [16.952133489480776]
  Confidence estimation is a task that aims to evaluate the trustworthiness of the model's prediction output during deployment. Previous works have outlined two important qualities that a reliable confidence estimation model should possess. We propose a meta-learning framework that can simultaneously improve upon both qualities in a confidence estimation model.
  arXiv  Detail & Related papers  (2022-10-13T06:34:23Z)
• Plex: Towards Reliability using Pretrained Large Model Extensions [69.13326436826227]
  We develop ViT-Plex and T5-Plex, pretrained large model extensions for vision and language modalities, respectively. Plex greatly improves the state-of-the-art across reliability tasks, and simplifies the traditional protocol. We demonstrate scaling effects over model sizes up to 1B parameters and pretraining dataset sizes up to 4B examples.
  arXiv  Detail & Related papers  (2022-07-15T11:39:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.