Retrieval-Augmented Linguistic Calibration

• URL: http://arxiv.org/abs/2605.19344v1
• Date: Tue, 19 May 2026 04:31:38 GMT
• Title: Retrieval-Augmented Linguistic Calibration
• Abstract summary: We model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct.<n>We present Retrieval-Augmented Linguistic truth (RALC), a lightweight post-hoc pipeline that propagates confidence signals back into natural language.
• Score: 57.41519309308438
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linguistic cues such as "I believe" and "probably" offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic confidence expressions remains underexplored. In particular, co-occurring linguistic cues, contextual variation, and subjective audience interpretation pose unique challenges. We therefore model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing interpretation variability that scalar representations discard. Within this distributional framework, we introduce faithfulness as a complementary evaluation dimension and present Faithfulness Divergence (FD), an information-theoretic metric quantifying the surprise induced in audience beliefs upon truth revelation. Building on these foundations, we present Retrieval-Augmented Linguistic Calibration (RALC), a lightweight post-hoc pipeline that propagates calibrated confidence signals back into natural language via retrieval-augmented rewriting. Across three QA benchmarks and five LLM families, RALC improves in-domain faithfulness and calibration up to 66% and 58%, respectively, outperforming black-box and grey-box calibration baselines.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Calibrating LLMs with Semantic-level Reward [35.81380656593668]
  We propose textbfCalibration with Semantic Reward (CSR), a framework that calibrates language models directly in semantic space without a verbalized confidence interface.<n>CSR consistently achieves lower ECE and higher AUROC than verbalized-confidence baselines across nearly all settings.
  arXiv  Detail & Related papers  (2026-05-15T03:55:11Z)
• Identifying Influential N-grams in Confidence Calibration via Regression Analysis [53.95767806659635]
  We identify what linguistic expressions are related to confidence by applying the regression method.<n>We show that large language models (LLMs) remain overconfident when reasoning is involved.<n>We reveal that confidence calibration is possible by simply suppressing those overconfident expressions without drops in performance.
  arXiv  Detail & Related papers  (2026-04-07T11:58:49Z)
• Closing the Confidence-Faithfulness Gap in Large Language Models [3.122242683664973]
  Large language models verbalize confidence scores that are largely detached from their actual accuracy.<n>We present a mechanistic interpretability analysis of verbalized confidence.<n>We introduce a two-stage adaptive steering pipeline that reads the model's internal accuracy estimate and steers verbalized output to match it.
  arXiv  Detail & Related papers  (2026-03-26T05:42:04Z)
• Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessment [8.300738063140129]
  We introduce a rubric-guided reasoning framework that explicitly encodes multi-aspect human assessment criteria.<n>We develop an uncertainty-calibrated regression approach supported by conformal calibration for interpretable confidence intervals.<n>Our results demonstrate that rubric-guided, uncertainty-calibrated reasoning offers a principled path toward trustworthy and explainable SpeechLLM-based speech assessment.
  arXiv  Detail & Related papers  (2026-02-28T11:08:24Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction [55.04308051033549]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a dominant paradigm for enhancing Large Language Models (LLMs) reasoning.<n>We introduceVerifier-Independent Curriculum Reinforcement Learning (VI-CuRL), a framework that leverages the model's intrinsic confidence to construct a curriculum independent from external verifiers.
  arXiv  Detail & Related papers  (2026-02-13T03:40:52Z)
• Can Large Language Models Express Uncertainty Like Human? [71.27418419522884]
  We release the first diverse, large-scale dataset of hedging expressions with human-annotated confidence scores.<n>We conduct the first systematic study of linguistic confidence across modern large language models.
  arXiv  Detail & Related papers  (2025-09-29T02:34:30Z)
• MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs [66.14178164421794]
  We introduce MetaFaith, a novel prompt-based calibration approach inspired by human metacognition.<n>We show that MetaFaith robustly improves faithful calibration across diverse models and task domains, enabling up to 61% improvement in faithfulness.
  arXiv  Detail & Related papers  (2025-05-30T17:54:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.