Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator

• URL: http://arxiv.org/abs/2505.16690v1
• Date: Thu, 22 May 2025 13:55:39 GMT
• Title: Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator
• Authors: Beier Luo, Shuoyuan Wang, Yixuan Li, Hongxin Wei,
• Abstract summary: Post-trained language models (PoLMs) often suffer from over-confidence, assigning high confidence to both correct and incorrect outputs.<n>A major obstacle in calibrating PoLMs is the scarcity of labeled data for individual downstream tasks.<n>We propose Disagreement-Aware Confidence Alignment (DACA) to optimize parameters in post-hoc confidence calibration.
• Score: 20.597317601065605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training of large language models is essential for adapting pre-trained language models (PLMs) to align with human preferences and downstream tasks. While PLMs typically exhibit well-calibrated confidence, post-trained language models (PoLMs) often suffer from over-confidence, assigning high confidence to both correct and incorrect outputs, which can undermine reliability in critical applications. A major obstacle in calibrating PoLMs is the scarcity of labeled data for individual downstream tasks. To address this, we propose Disagreement-Aware Confidence Alignment (DACA), a novel unsupervised method to optimize the parameters (e.g., temperature $\tau$) in post-hoc confidence calibration. Our method is motivated by the under-confidence issue caused by prediction disagreement between the PLM and PoLM while aligning their confidence via temperature scaling. Theoretically, the PLM's confidence underestimates PoLM's prediction accuracy on disagreement examples, causing a larger $\tau$ and producing under-confident predictions. DACA mitigates this by selectively using only agreement examples for calibration, effectively decoupling the influence of disagreement. In this manner, our method avoids an overly large $\tau$ in temperature scaling caused by disagreement examples, improving calibration performance. Extensive experiments demonstrate the effectiveness of our method, improving the average ECE of open-sourced and API-based LLMs (e.g. GPT-4o) by up to 15.08$\%$ on common benchmarks.

Related papers

• Calibrating Uncertainty Quantification of Multi-Modal LLMs using Grounding [48.92310906093414]
  We introduce a novel approach for calibrating uncertainty quantification (UQ) tailored for multi-modal large language models (LLMs)<n>We leverage cross-modal consistency in addition to self-consistency to improve the calibration of the multi-modal models.<n>We evaluate the proposed approach across multiple multi-modal tasks, such as medical question answering (Slake) and visual question answering (VQAv2), considering multi-modal models such as LLaVA-Med and LLaVA.
  arXiv  Detail & Related papers  (2025-04-30T19:19:21Z)
• Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation [26.580361841501514]
  Vision-language models (VLMs) excel in various multimodal tasks but frequently suffer from poor calibration.<n>This miscalibration undermines user trust, especially when models confidently provide incorrect or fabricated information.<n>We propose a novel Confidence through Semantic Perturbation (CSP) framework to improve the calibration of verbalized confidence for object-centric queries.
  arXiv  Detail & Related papers  (2025-04-21T04:01:22Z)
• SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales [29.33581578047835]
  SaySelf is a training framework that teaches large language models to express more accurate fine-grained confidence estimates. In addition, SaySelf directs LLMs to produce self-reflective rationales that clearly identify gaps in their parametric knowledge. We show that the generated self-reflective rationales are reasonable and can further contribute to the calibration.
  arXiv  Detail & Related papers  (2024-05-31T16:21:16Z)
• Calibrating Large Language Models Using Their Generations Only [44.26441565763495]
  APRICOT is a method to set confidence targets and train an additional model that predicts an LLM's confidence based on its textual input and output alone. It is conceptually simple, does not require access to the target model beyond its output, does not interfere with the language generation, and has a multitude of potential usages. We show how our approach performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering to detect incorrect LLM answers.
  arXiv  Detail & Related papers  (2024-03-09T17:46:24Z)
• Calibrating Large Language Models with Sample Consistency [76.23956851098598]
  We explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. Results show that consistency-based calibration methods outperform existing post-hoc approaches. We offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
  arXiv  Detail & Related papers  (2024-02-21T16:15:20Z)
• Making Pre-trained Language Models both Task-solvers and Self-calibrators [52.98858650625623]
  Pre-trained language models (PLMs) serve as backbones for various real-world systems. Previous work shows that introducing an extra calibration task can mitigate this issue. We propose a training algorithm LM-TOAST to tackle the challenges.
  arXiv  Detail & Related papers  (2023-07-21T02:51:41Z)
• Certified Robustness for Large Language Models with Self-Denoising [42.916661225753145]
  We propose to denoise the corrupted inputs with large language models (LLMs) in a self-denoising manner. Our method outperforms the existing certification methods under both certified robustness and empirical robustness.
  arXiv  Detail & Related papers  (2023-07-14T05:40:24Z)
• Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback [91.22679548111127]
  A trustworthy real-world prediction system should produce well-calibrated confidence scores. We show that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities.
  arXiv  Detail & Related papers  (2023-05-24T10:12:33Z)
• A Close Look into the Calibration of Pre-trained Language Models [56.998539510508515]
  Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. We study the dynamic change in PLMs' calibration performance in training. We extend two recently proposed learnable methods that directly collect data to train models to have reasonable confidence estimations.
  arXiv  Detail & Related papers  (2022-10-31T21:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.