Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration

• URL: http://arxiv.org/abs/2607.13753v2
• Date: Mon, 20 Jul 2026 09:24:51 GMT
• Title: Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration
• Abstract summary: We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation.<n>We show that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation.<n>We propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals.
• Score: 20.813380812371637
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models have made strong reasoning gains through supervised fine-tuning, reinforcement learning, and on-policy distillation, yet these post-training methods are usually evaluated only by final-answer accuracy. We study how they reshape confidence during reasoning. We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation, corresponding to difficulty estimation, early termination, and answer aggregation. Through a controlled comparison on mathematical reasoning benchmarks, we find that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation. We further show that confidence reliability is position-dependent: RL confidence becomes informative after a path-commitment phase, while OPD confidence is useful early but can become inversely calibrated later. Based on this observation, we propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf improves RL answer aggregation by 6.1 points over majority voting and consistently improves OPD early stopping under tight token budgets, with gains up to 4.3 points by avoiding its later inverse-calibration region, showing that \emph{confidence in reasoning models should be used both stage-wise and position-awarely}. Our code is available at https://github.com/EIT-NLP/Post-Training-Calibration.

Related papers

• Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning [81.57028614960576]
  We show that the Confidence Region possesses two critical properties: 1) High Reliability -- answers in the confidence region become highly accurate and stable, and 2) High Redundancy -- models generate unnecessary tokens long after reaching the correct answer.<n>These properties unlock more efficient and reliable inference strategies.
  arXiv  Detail & Related papers  (2026-06-01T10:11:14Z)
• Enhancing LLM Metacognition via Cognitive Pairwise Training [56.19495984013656]
  Reinforcement learning with verifiable rewards can make models more willing to give confident answers when evidence or reasoning is unreliable.<n>Existing SFT or RL methods mainly teach LLMs to refuse or express uncertainty at the response level.<n>We propose Cognitive Pairwise Training (CPT), a cognitive mid-training alignment stage that turns pairwise comparisons over reasoning traces into a reusable alignment signal.
  arXiv  Detail & Related papers  (2026-05-30T19:53:19Z)
• Are We Overconfident in Models and Results for Semi-Supervised 3D Medical Image Segmentation? [4.120238673372104]
  Semi-supervised learning has become a dominant paradigm for reducing annotation costs.<n>We argue that the current progress is clouded by a twofold overconfidence problem.<n>We propose a tri-space principled segmentation framework founded on a dual-axis reliability assessment engine.
  arXiv  Detail & Related papers  (2026-05-25T08:16:40Z)
• Understanding and Mitigating Premature Confidence for Better LLM Reasoning [76.16007941549857]
  Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps.<n>We find such a signal in how the model's confidence evolves during reasoning.<n>We exploit this in progressive confidence shaping, a reinforcement learning objective that trains models to update their confidence as they reason rather than commit early.
  arXiv  Detail & Related papers  (2026-05-23T04:42:45Z)
• Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning [5.094079537924077]
  We show that confidence trajectories separate correct from incorrect reasoning traces.<n> correctness-related information is enriched in the tail of reasoning.<n>We propose NeuralConf, a lightweight estimator that learns from confidence trajectories for correctness evaluation.
  arXiv  Detail & Related papers  (2026-05-16T05:57:00Z)
• Confidence-Aware Alignment Makes Reasoning LLMs More Reliable [65.44962502963378]
  We introduce CASPO, a framework that aligns token-level confidence with step-wise logical correctness without training a separate reward model.<n>During inference, we propose Confidence-aware Thought (CaT), which dynamically prunes uncertain reasoning branches with negligible O(V) latency.<n>Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency.
  arXiv  Detail & Related papers  (2026-05-08T07:08:25Z)
• Process Supervision of Confidence Margin for Calibrated LLM Reasoning [52.373121066425455]
  Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability.<n>Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation.<n>We introduce Reinforcement Learning with Confidence Margin (textbfRLCM), a calibration-aware RL framework that jointly optimize correctness and confidence reliability.
  arXiv  Detail & Related papers  (2026-04-25T14:40:13Z)
• Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection [0.0]
  Large language models (LLMs) are increasingly deployed in critical decision-making systems.<n>We introduce a normalized confidence score based on output anchor token probabilities.<n>This enables direct detection of errors and hallucinations with minimal overhead.
  arXiv  Detail & Related papers  (2026-02-18T07:05:12Z)
• Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation [63.49409574310576]
  Large language models (LLMs) exhibit overconfidence, assigning high confidence scores to incorrect predictions.<n>We introduce FineCE, a novel confidence estimation method that delivers accurate, fine-grained confidence scores during text generation.<n>Our code and all baselines used in the paper are available on GitHub.
  arXiv  Detail & Related papers  (2025-08-16T13:29:35Z)
• SteerConf: Steering LLMs for Confidence Elicitation [11.872504642312705]
  Large Language Models (LLMs) exhibit impressive performance across diverse domains but often suffer from overconfidence.<n>We propose SteerConf, a novel framework that systematically steers LLMs' confidence scores to improve their calibration and reliability.
  arXiv  Detail & Related papers  (2025-03-04T18:40:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.