Confident at the moment of action: belief miscalibration in LLM play under hidden information

• URL: http://arxiv.org/abs/2608.24691v1
• Date: Tue, 25 Aug 2026 15:18:18 GMT
• Title: Confident at the moment of action: belief miscalibration in LLM play under hidden information
• Abstract summary: Agentic systems increasingly gate actions on a model's own stated confidence, which assumes confidence tracks correctness at the moment of acting.<n>We test this in a hidden-information chess variant where royal status can be secretly, repeatedly relocated between pieces.<n>A model exhibiting this pattern can still win the game its belief was about, which is why outcome-only evaluation would not detect it.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agentic systems increasingly gate actions on a model's own stated confidence, which assumes confidence tracks correctness at the moment of acting. We test this in a hidden-information chess variant where royal status can be secretly, repeatedly relocated between pieces, and where an agent's stated probability distribution over the opponent's hidden royal piece -- elicited every turn, separately from the move it chooses -- is scored against ground truth recoverable after the game. Across two independent batches, captures made at high stated confidence ($\geq 0.5$) about the hidden piece's location were correct in 1 of 62 cases. The calibration deficit is concentrated almost entirely in these events: 99.3% of it in the original batch, 98.7% in the replication. The same pattern, in weaker form, orders consistently (point estimates only; most pairwise gaps are not statistically distinguishable at this sample size) across four further model configurations spanning a second provider -- reported as scope for the finding, not as evidence that capability predicts calibration: a same-model comparison at a fixed external leaderboard score shows a deliberation-budget change alone moves the metric by nearly as much as a large cross-model gap. In a separate seat, conventional evaluation axes -- legality, cost, latency, completion rate -- can dissociate entirely from belief quality, with the configuration winning on every conventional axis producing the worst belief quality tested. A model exhibiting this pattern can still win the game its belief was about, which is why outcome-only evaluation would not detect it.

Related papers

• When Confidence Signals Disagree: Local and Global Confidence in Autoregressive Language Models [0.0]
  We investigate whether different confidence readouts are empirically interchangeable in an autoregressive language model.<n>Across MMLU and ARC Challenge, the two signals are weakly correlated and differ substantially in their association with correctness.<n>Larger local--global confidence gaps are associated with higher answer entropy, more distinct sampled answers, and lower modal-answer concentration.
  arXiv  Detail & Related papers  (2026-09-15T10:08:48Z)
• Reliability analysis for BraTS-GoAT segmentation: a controlled robustness study of deep-ensemble uncertainty [5.504955093712013]
  Deep networks segment brain tumours accurately in-distribution, but can fail silently when the input differs from their training data.<n>We ask how well a model segments, and whether its uncertainty knows when it is wrong.
  arXiv  Detail & Related papers  (2026-08-13T13:27:21Z)
• When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals [0.0]
  LLM-as-judge is increasingly the default for evaluating AI systems in enterprise pipelines.<n>We show that consistency -- agreement among judges, or a model's own samples -- indicates correctness.<n>We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study.
  arXiv  Detail & Related papers  (2026-07-09T02:46:51Z)
• Verifiable Rewards for Calibrated Probabilistic Forecasting [3.77661940418406]
  Reinforcement learning with rewards can train calibrated probabilistic forecasters.<n>We study aleatoric forecasting, where the forecast itself is the calibrated output and the label is one outcome.<n>We introduce a verifiable, label-free reward, a state-conditioned empirical win rate estimated from past outcomes.
  arXiv  Detail & Related papers  (2026-06-30T20:42:45Z)
• LLMs Show No Signs Of Individuated Metacognition [0.023227405857540805]
  We decompose binary confidence judgements from 20 Large Language Models.<n>We ask whether two models that differ in confidence also differ in performance.<n>We find no evidence for significant verbalised individuated metacognition in any tested domain.
  arXiv  Detail & Related papers  (2026-05-22T23:54:33Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• Belief Memory: Agent Memory Under Partial Observability [56.41506249481312]
  We propose BeliefMem, which shifts the memory paradigm from committing to a single conclusion per observation to retaining multiple candidate conclusions with their probabilities.<n> BeliefMem preserves the uncertainty that the deterministic paradigm discards, enabling the agent to act with high confidence.<n> Empirical evaluations on LoCoMo and ALFWorld benchmarks show that, even with limited data, BeliefMem achieves the best average performance.
  arXiv  Detail & Related papers  (2026-05-07T02:03:13Z)
• The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation [67.26315138466312]
  We propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts.<n>Our findings highlight that capability distillation does not imply confidence, and that confidence should be treated as an essential objective in post-training.
  arXiv  Detail & Related papers  (2026-04-18T04:43:40Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Beyond calibration: estimating the grouping loss of modern neural networks [68.8204255655161]
  Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings.
  arXiv  Detail & Related papers  (2022-10-28T07:04:20Z)
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification [58.03725169462616]
  We show theoretically that over-parametrization is not the only reason for over-confidence. We prove that logistic regression is inherently over-confident, in the realizable, under-parametrized setting. Perhaps surprisingly, we also show that over-confidence is not always the case.
  arXiv  Detail & Related papers  (2021-02-15T21:38:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.