Do Reasoning Models Show Better Verbalized Calibration?

• URL: http://arxiv.org/abs/2504.06564v1
• Date: Wed, 09 Apr 2025 03:58:19 GMT
• Title: Do Reasoning Models Show Better Verbalized Calibration?
• Authors: Qingcheng Zeng, Weihao Xuan, Leyang Cui, Rob Voigt,
• Abstract summary: We investigate the calibration properties of LRMs trained via supervised fine-tuning distillation on long reasoning traces.<n>Our findings reveal that LRMs significantly outperform instruction-tuned models on complex reasoning tasks in both accuracy and confidence calibration.<n>Our results provide evidence for a potentially critical role of reasoning-oriented RL training in improving LLMs' capacity for generating trustworthy, self-aware outputs.
• Score: 19.776645881640178
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large reasoning models (LRMs) have recently shown impressive capabilities in complex reasoning by leveraging increased test-time computation and exhibiting behaviors akin to human-like deliberation. Despite these advances, it remains an open question whether LRMs are better calibrated - particularly in their verbalized confidence - compared to instruction-tuned counterparts. In this paper, we investigate the calibration properties of LRMs trained via supervised fine-tuning distillation on long reasoning traces (henceforth SFT reasoning models) and outcome-based reinforcement learning for reasoning (henceforth RL reasoning models) across diverse domains. Our findings reveal that LRMs significantly outperform instruction-tuned models on complex reasoning tasks in both accuracy and confidence calibration. In contrast, we find surprising trends in the domain of factuality in particular. On factuality tasks, while Deepseek-R1 shows strong calibration behavior, smaller QwQ-32B shows no improvement over instruct models; moreover, SFT reasoning models display worse calibration (greater overconfidence) compared to instruct models. Our results provide evidence for a potentially critical role of reasoning-oriented RL training in improving LLMs' capacity for generating trustworthy, self-aware outputs.

Related papers

• Exploring and Exploiting the Inherent Efficiency within Large Reasoning Models for Self-Guided Efficiency Enhancement [101.77467538102924]
  Large reasoning models (LRMs) exhibit overthinking, which hinders efficiency and inflates inference cost.<n>We propose two lightweight methods to enhance LRM efficiency.<n>First, we introduce Efficiency Steering, a training-free activation steering technique that modulates reasoning behavior via a single direction.<n>Second, we develop Self-Rewarded Efficiency RL, a reinforcement learning framework that dynamically balances task accuracy and brevity.
  arXiv  Detail & Related papers  (2025-06-18T17:18:12Z)
• From Emergence to Control: Probing and Modulating Self-Reflection in Language Models [23.176641726866105]
  Self-reflection is a powerful behavior enabled by reinforcement learning with verifiable rewards.<n>We show that self-reflection is not exclusive to fine-tuned models.
  arXiv  Detail & Related papers  (2025-06-13T20:40:13Z)
• Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability [70.4107059502882]
  Training language models with rationales augmentation has been shown to be beneficial in many existing works.<n>We conduct comprehensive investigations to thoroughly inspect the impact of rationales on model performance.
  arXiv  Detail & Related papers  (2025-05-30T02:39:37Z)
• Can Large Reasoning Models Self-Train? [58.953117118687096]
  Scaling the performance of large language models increasingly depends on methods that reduce reliance on human supervision.<n>We propose an online self-training reinforcement learning algorithm that leverages the model's self-consistency to infer correctness signals and train without any ground-truth supervision.
  arXiv  Detail & Related papers  (2025-05-27T17:16:00Z)
• Reasoning Models Better Express Their Confidence [33.72935464539185]
  Large language models (LLMs) often fail to communicate their confidence accurately, making it difficult to assess when they might be wrong and limiting their reliability.<n>In this work, we demonstrate that reasoning models-LLMs that engage in extended chain-of-thought (CoT) reasoning-exhibit superior performance not only in problem-solving but also in accurately expressing their confidence.
  arXiv  Detail & Related papers  (2025-05-20T15:19:00Z)
• SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models [39.551767637896404]
  This work revisits the dominant supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm for training Large Vision-Language Models (LVLMs) We show that SFT can significantly undermine subsequent RL by inducing pseudo reasoning paths'' imitated from expert models. We introduce VLAA-Thinking, a new multimodal dataset designed to support reasoning in LVLMs.
  arXiv  Detail & Related papers  (2025-04-10T16:54:05Z)
• Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuning [12.559028963968247]
  We investigate the crucial relationship between a model's reasoning ability and fairness.<n>We find that larger models with stronger reasoning abilities exhibit substantially lower stereotypical bias.<n>We introduce ReGiFT, a novel approach that extracts structured reasoning traces from advanced reasoning models and infuses them into models that lack such capabilities.
  arXiv  Detail & Related papers  (2025-04-08T03:21:51Z)
• Concise Reasoning via Reinforcement Learning [13.657506042120167]
  We revisit the core principles of reinforcement learning (RL)<n>We uncover a natural correlation between conciseness and accuracy that has been largely overlooked.<n>We show that introducing a secondary phase of RL post-training, using a small set of problems and limited resources, can significantly reduce a model's chain of thought.
  arXiv  Detail & Related papers  (2025-04-07T15:35:54Z)
• SEAL: Steerable Reasoning Calibration of Large Language Models for Free [58.190800043449336]
  Large Language Models (LLMs) have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism. Recent studies reveal substantial redundancy in the CoT reasoning traces, which negatively impacts model performance. We introduce SEAL, a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains.
  arXiv  Detail & Related papers  (2025-04-07T02:42:07Z)
• ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with Iterative Retrieval Augmented Generation [38.64751082999587]
  Large Reasoning Models (LRMs) exhibit remarkable reasoning abilities but rely primarily on parametric knowledge, limiting factual accuracy.<n>We propose ReaRAG, a factuality-enhanced reasoning model that explores diverse queries without excessive iterations.<n>Our study enhances LRMs' factuality while effectively integrating robust reasoning for Retrieval-Augmented Generation (RAG)
  arXiv  Detail & Related papers  (2025-03-27T17:44:18Z)
• R-PRM: Reasoning-Driven Process Reward Modeling [53.06844294668382]
  Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step. Existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy. We propose Reasoning-Driven Process Reward Modeling (R-PRM) R-PRM generates seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-27T09:23:08Z)
• OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning via Iterative Self-Improvement [91.88062410741833]
  This study investigates whether similar reasoning capabilities can be successfully integrated into large vision-language models (LVLMs)<n>We consider an approach that iteratively leverages supervised fine-tuning (SFT) on lightweight training data and Reinforcement Learning (RL) to further improve model generalization.<n>OpenVLThinker, a LVLM exhibiting consistently improved reasoning performance on challenging benchmarks such as MathVista, MathVerse, and MathVision, demonstrates the potential of our strategy for robust vision-language reasoning.
  arXiv  Detail & Related papers  (2025-03-21T17:52:43Z)
• Improve Vision Language Model Chain-of-thought Reasoning [86.83335752119741]
  Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. We show that training VLM on short answers does not generalize well to reasoning tasks that require more detailed responses.
  arXiv  Detail & Related papers  (2024-10-21T17:00:06Z)
• Calibrating Large Language Models with Sample Consistency [76.23956851098598]
  We explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. Results show that consistency-based calibration methods outperform existing post-hoc approaches. We offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
  arXiv  Detail & Related papers  (2024-02-21T16:15:20Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• Crystal: Introspective Reasoners Reinforced with Self-Feedback [118.53428015478957]
  We propose a novel method to develop an introspective commonsense reasoner, Crystal. To tackle commonsense problems, it first introspects for knowledge statements related to the given question, and subsequently makes an informed prediction that is grounded in the previously introspected knowledge. Experiments show that Crystal significantly outperforms both the standard supervised finetuning and chain-of-thought distilled methods, and enhances the transparency of the commonsense reasoning process.
  arXiv  Detail & Related papers  (2023-10-07T21:23:58Z)
• Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning [76.98542249776257]
  Large-scale language models often face the challenge of "hallucination" We introduce an uncertainty-aware in-context learning framework to empower the model to enhance or reject its output in response to uncertainty.
  arXiv  Detail & Related papers  (2023-10-07T12:06:53Z)
• The Confidence-Competence Gap in Large Language Models: A Cognitive Study [3.757390057317548]
  Large Language Models (LLMs) have acquired ubiquitous attention for their performances across diverse domains. We exploit these models with diverse sets of questionnaires and real-world scenarios. Our findings reveal intriguing instances where models demonstrate high confidence even when they answer incorrectly.
  arXiv  Detail & Related papers  (2023-09-28T03:50:09Z)
• CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration [59.48235003469116]
  We show that data augmentation consistently enhances OOD performance. We also show that CF augmented models which are easier to calibrate also exhibit much lower entropy when assigning importance.
  arXiv  Detail & Related papers  (2023-09-14T16:16:40Z)
• Does Self-Rationalization Improve Robustness to Spurious Correlations? [19.553357015260687]
  We ask whether training models to self-rationalize can aid in their learning to solve tasks for the right reasons. We evaluate robustness to spurious correlations in fine-tuned encoder-decoder and decoder-only models of six different sizes. We find that while self-rationalization can improve robustness to spurious correlations in low-resource settings, it tends to hurt robustness in higher-resource settings.
  arXiv  Detail & Related papers  (2022-10-24T19:54:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.