Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?

• URL: http://arxiv.org/abs/2605.28778v1
• Date: Wed, 27 May 2026 17:38:00 GMT
• Title: Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?
• Abstract summary: It remains unclear whether models can apply their own linguistic confidence framework to associate markers with specific confidence levels in a stable and generalizable way.<n>We formalize _marker internal confidence_ (MIC) as the estimated intrinsic confidence a model associates with a specific epistemic marker in a given task domain.<n>Applying our analysis framework to diverse models and tasks, we find that LLMs remain faithfully miscalibrated even under model-centric interpretation of marker meanings.
• Score: 50.17126900886782
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLMs' linguistically expressed confidence should faithfully reflect their intrinsic uncertainty. While recent work shows LLMs struggle to use epistemic markers (e.g., "it is likely...") in a human-aligned fashion, it remains unclear whether models can apply their own linguistic confidence framework to associate markers with specific confidence levels in a stable and generalizable way, and how contextual features impact this ability. We conduct the first systematic study of this question, formalizing _marker internal confidence_ (MIC) as the estimated intrinsic confidence a model associates with a specific epistemic marker in a given task domain. We present 7 metrics to evaluate the stability of MICs within and across distributions. Applying our analysis framework to diverse models and tasks, we find that LLMs remain faithfully miscalibrated even under model-centric interpretation of marker meanings, struggling to differentiate markers by internal confidence across distributions despite preserving a somewhat consistent ranking order across tasks. This supplies critical, complementary evidence to existing work toward a holistic understanding of faithful calibration in LLMs, emphasizing the need for more aligned and stable marker use to improve trustworthiness and reliability.

Related papers

• Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness [4.129847064263056]
  We systematically evaluate the performance of Large Language Models for rubric-based short-answer grading.<n>We find that alignment is strong for binary tasks but degrades with increased rubric granularity.<n>Experiments reveal that while the model is resilient to prompt injection, it is sensitive to synonym substitutions.
  arXiv  Detail & Related papers  (2025-12-21T05:22:04Z)
• MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs [66.14178164421794]
  We introduce MetaFaith, a novel prompt-based calibration approach inspired by human metacognition.<n>We show that MetaFaith robustly improves faithful calibration across diverse models and task domains, enabling up to 61% improvement in faithfulness.
  arXiv  Detail & Related papers  (2025-05-30T17:54:08Z)
• Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty? [46.28794610965004]
  Large language models (LLMs) are increasingly used in high-stakes domains.<n>It remains unclear whether LLMs consistently use these markers to reflect their intrinsic confidence.<n>We first define marker confidence as the observed accuracy when a model employs an epistemic marker.<n>Our results show that while markers generalize well within the same distribution, their confidence is inconsistent in out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-05-30T16:41:24Z)
• On Verbalized Confidence Scores for LLMs [25.160810008907397]
  Uncertainty quantification for large language models (LLMs) can establish more human trust into their responses.<n>This work focuses on asking the LLM itself to verbalize its uncertainty with a confidence score as part of its output tokens.<n>We assess the reliability of verbalized confidence scores with respect to different datasets, models, and prompt methods.
  arXiv  Detail & Related papers  (2024-12-19T11:10:36Z)
• Learning to Route LLMs with Confidence Tokens [43.63392143501435]
  Large language models (LLMs) have demonstrated impressive performance on several tasks and are increasingly deployed in real-world applications.<n>In high-stakes settings, it becomes vital to know when the output of an LLM may be unreliable.<n>We study the extent to which LLMs can reliably indicate confidence in their answers, and how this notion of confidence can translate into downstream accuracy gains.
  arXiv  Detail & Related papers  (2024-10-17T07:28:18Z)
• When to Trust LLMs: Aligning Confidence with Response Quality [49.371218210305656]
  We propose CONfidence-Quality-ORDer-preserving alignment approach (CONQORD) It integrates quality reward and order-preserving alignment reward functions. Experiments demonstrate that CONQORD significantly improves the alignment performance between confidence and response accuracy.
  arXiv  Detail & Related papers  (2024-04-26T09:42:46Z)
• TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness [58.721012475577716]
  Large Language Models (LLMs) have demonstrated impressive capabilities across various domains, prompting a surge in their practical applications. This paper introduces TrustScore, a framework based on the concept of Behavioral Consistency, which evaluates whether an LLMs response aligns with its intrinsic knowledge.
  arXiv  Detail & Related papers  (2024-02-19T21:12:14Z)
• What Large Language Models Know and What People Think They Know [13.939511057660013]
  Large language models (LLMs) are increasingly integrated into decision-making processes.<n>To earn human trust, LLMs must be well calibrated so that they can accurately assess and communicate the likelihood of their predictions being correct.<n>Here we explore the calibration gap, which refers to the difference between human confidence in LLM-generated answers and the models' actual confidence, and the discrimination gap, which reflects how well humans and models can distinguish between correct and incorrect answers.
  arXiv  Detail & Related papers  (2024-01-24T22:21:04Z)
• TrustLLM: Trustworthiness in Large Language Models [446.5640421311468]
  This paper introduces TrustLLM, a comprehensive study of trustworthiness in large language models (LLMs) We first propose a set of principles for trustworthy LLMs that span eight different dimensions. Based on these principles, we establish a benchmark across six dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics.
  arXiv  Detail & Related papers  (2024-01-10T22:07:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.