Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs

• URL: http://arxiv.org/abs/2604.10585v1
• Date: Sun, 12 Apr 2026 11:09:49 GMT
• Title: Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
• Abstract summary: We investigate whether sycophantic reward signals degrade calibration.<n>Post-hoc matrix scaling applied to all three models reduces ECE by $40$--$64%$ and improves accuracy by $1.5$--$3.0$ percentage points.<n>These findings establish a methodology for evaluating the calibration impact of reward hacking and motivate calibration-aware training objectives.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern large language models (LLMs) are increasingly fine-tuned via reinforcement learning from human feedback (RLHF) or related reward optimisation schemes. While such procedures improve perceived helpfulness, we investigate whether sycophantic reward signals degrade calibration -- a property essential for reliable uncertainty quantification. We fine-tune Qwen3-8B under three regimes: no fine-tuning (base), neutral supervised fine-tuning (SFT) on TriviaQA, and sycophancy-inducing Group Relative Policy Optimisation (GRPO) that rewards agreement with planted wrong answers. Evaluating on $1{,}000$ MMLU items across five subject domains with bootstrap confidence intervals and permutation testing, we find that \textbf{sycophantic GRPO produces consistent directional calibration degradation} -- ECE rises by $+0.006$ relative to the base model and MCE increases by $+0.010$ relative to neutral SFT -- though the effect does not reach statistical significance ($p = 0.41$) at this training budget. Post-hoc matrix scaling applied to all three models reduces ECE by $40$--$64\%$ and improves accuracy by $1.5$--$3.0$ percentage points. However, the sycophantic model retains the highest post-scaling ECE relative to the neutral SFT control ($0.042$ vs.\ $0.037$), suggesting that reward-induced miscalibration leaves a structured residual even after affine correction. These findings establish a methodology for evaluating the calibration impact of reward hacking and motivate calibration-aware training objectives.

Related papers

• Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization [60.89431018071735]
  Reinforcement learning with group-relative advantages has become the de facto standard for post-training language model reasoners.<n>We show that rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation.<n>We introduce saturation-aware reweighting, which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation.
  arXiv  Detail & Related papers  (2026-08-17T04:07:50Z)
• Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models [0.0]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving mathematical reasoning in language models.<n>We systematically compare five reward conditions applied to Qwen2.5-0.5B fine-tuned with Group Relative Policy Optimization (GRPO) on GSM8K.<n>Our results demonstrate that reward derivation is a first-order design decision for RLVR, with process-level supervision substantially improving both accuracy and trace fidelity in small language models.
  arXiv  Detail & Related papers  (2026-07-03T02:16:06Z)
• Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling [2.8793525226737327]
  A novel reinforcement learning algorithm integrates correctness, calibration and dataset-informed reference accuracy rewards together.<n>C3RL enhances calibration without sacrificing accuracy, outperforming the current state-of-the-art method in both performance and calibration metrics.<n>Experiments show that CAS surpasses majority voting on both in-domain and out-of-domain datasets while reducing the inference budget by up to 12.33 times.
  arXiv  Detail & Related papers  (2026-07-02T02:29:33Z)
• Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes [53.726365933748134]
  We analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions.<n>We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above $sim$70% activation sparsity.
  arXiv  Detail & Related papers  (2026-05-17T21:29:20Z)
• Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret [71.69884486156359]
  We study episodic RL from emphmulti-source imperfect preferences through a cumulative imperfection budget.<n>We propose a unified algorithm with regret $tildeO(sqrtK/M+)$, which exhibits a best-of-both-regimes behavior.
  arXiv  Detail & Related papers  (2026-03-20T19:34:53Z)
• REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge [83.2858110368572]
  We propose textbfREAL (underlineREgression-underlineAware Reinforcement underlineLThought), a principled RL framework designed to optimize regression rewards.<n>We show that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods.
  arXiv  Detail & Related papers  (2026-03-17T21:19:08Z)
• $V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts [81.48669089692189]
  Generalist Value Models (such as $V_0.5$) achieve pre-trained value estimation by explicitly encoding model capabilities in-context.<n>We propose $V_0.5$, which adaptively fuses the baseline predicted by such value model with the empirical mean derived from sparse rollouts.<n>$V_0.5$ significantly outperforms GRPO and DAPO, achieving faster convergence and over some 10% performance improvement.
  arXiv  Detail & Related papers  (2026-03-11T14:57:41Z)
• Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models [68.45272703833209]
  We show that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.<n>We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities.<n>We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.
  arXiv  Detail & Related papers  (2026-02-20T23:38:03Z)
• Complex-Valued Unitary Representations as Classification Heads for Improved Uncertainty Quantification in Deep Neural Networks [1.932555230783329]
  Modern deep neural networks achieve high predictive accuracy but remain poorly calibrated.<n>We propose a quantum-inspired classification head architecture that projects backbone features into a complex-valued space.<n>Our ablation study on CIFAR-10 reveals that the unitary magnitude head (complex features evolved under a Cayley unitary, read out via magnitude and softmax) achieves an Expected Error (ECE) of 0.0146, representing a 2.4x improvement over a standard softmax head.<n>Surprisingly, replacing the softmax readout with a Hilbert rule measurement layer - the quantum-mechanically motivated approach - degrades calibration to an ECE of 0.0819
  arXiv  Detail & Related papers  (2026-02-17T00:45:27Z)
• CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training [73.46600457802693]
  We introduce a new method that counteracts the loss induced by quantization.<n>CAGE significantly improves upon the state-of-theart methods in terms of accuracy, for similar computational cost.<n>For QAT pre-training of Llama models, CAGE matches the accuracy achieved at 4-bits (W4A4) with the prior best method.
  arXiv  Detail & Related papers  (2025-10-21T16:33:57Z)
• OrthoGrad Improves Neural Calibration [0.0]
  $perp$Grad constrains descent directions to address overconfidence.<n>$perp$Grad is a geometry-aware modification to optimization.
  arXiv  Detail & Related papers  (2025-06-04T22:12:46Z)
• Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation models [3.938980910007962]
  Confidence calibration is an emerging challenge in real-world decision systems based on foundations models.<n>We propose a penalty incorporated into loss objective that penalizes incorrect classifications whenever one is made during finetuning.<n>We refer to it as textitconfidence misalignment penalty (CMP).
  arXiv  Detail & Related papers  (2025-01-29T11:54:37Z)
• GAQAT: gradient-adaptive quantization-aware training for domain generalization [54.31450550793485]
  We propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG.<n>Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization.<n>Extensive experiments validate the effectiveness of the proposed GAQAT framework.
  arXiv  Detail & Related papers  (2024-12-07T06:07:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.