What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

• URL: http://arxiv.org/abs/2607.08046v1
• Date: Thu, 09 Jul 2026 01:52:09 GMT
• Title: What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness
• Abstract summary: We train representation-pooling probes on intermediate activations and find they achieve substantially better calibration.<n>We assess chain-of-thought faithfulness through evidence ablation and diversionary injection.<n>A single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens.
• Score: 3.4789212936038116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with Eternis-Forecaster 8B on OpenForesight, we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM-4.7-Flash and GLM-4.5-Air. We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model's forecast while leaving the reasoning trace untouched. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84% of cases, including when the CoT conceals the perturbation's influence. Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens, with no loss of accuracy. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly.

Related papers

• Training Large Language Models for Self-Explanation Faithfulness [10.933553638236036]
  We propose a Reinforcement Learning (RL) method to directly optimize the faithfulness of self-explanations.<n>We investigate whether models can be trained to accurately detect factors that affect their decisions.<n>We show that models can be trained to implicitly identify influential factors and disclose them.
  arXiv  Detail & Related papers  (2026-07-23T09:20:38Z)
• The Knowing-Saying Gap: When Probes See Errors that Confidence Misses [20.46211358172975]
  Linear probes detect corrupted context in language models with near-perfect accuracy, yet this does not translate into reliable failure prediction.<n>Across multi-hop arithmetic chains, probes that detect corruption turn out to be uninformative about final answer correctness.<n>As a real-time monitor, probe-based interventions are sharply model and error-type dependent.
  arXiv  Detail & Related papers  (2026-07-21T12:13:10Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• Understanding and Mitigating Premature Confidence for Better LLM Reasoning [76.16007941549857]
  Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps.<n>We find such a signal in how the model's confidence evolves during reasoning.<n>We exploit this in progressive confidence shaping, a reinforcement learning objective that trains models to update their confidence as they reason rather than commit early.
  arXiv  Detail & Related papers  (2026-05-23T04:42:45Z)
• Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought [11.955186033088351]
  We provide evidence of performative chain-of-thought (CoT) in reasoning models.<n>We compare activation probing, early forced answering, and a CoT monitor across two large models.<n>We contrast this with genuine reasoning in difficult multihop GPQA-Diamond questions.
  arXiv  Detail & Related papers  (2026-03-05T18:55:16Z)
• How Robustly do LLMs Understand Execution Semantics? [3.2717315277334706]
  LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains open.<n>We study LLMs through the lens of robustness of their code understanding using a standard program-output prediction task.<n>Our findings both point to limitations in the way all models understand code, and establish the value of using perturbation to evaluate code models.
  arXiv  Detail & Related papers  (2026-02-24T19:07:25Z)
• Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution [79.98699884805636]
  Reasoning Execution by Multiple Listeners (REMUL) is a multi-party reinforcement learning approach.<n>REMUL builds on the hypothesis that reasoning traces which other parties can follow will be more faithful.<n>Speakers are rewarded for producing reasoning that is clear to listeners.
  arXiv  Detail & Related papers  (2026-02-18T02:55:55Z)
• Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuits [17.17286544824839]
  Large language models (LLMs) generate fluent and complex outputs but often fail to recognize their own mistakes and hallucinations.<n>We introduce Gnosis, a lightweight self-awareness mechanism that enables frozen LLMs to perform intrinsic self-verification.<n>Gnosis consistently outperforms strong internal baselines and large external judges in both accuracy and calibration.
  arXiv  Detail & Related papers  (2025-12-23T18:21:32Z)
• Verifying Large Language Models' Reasoning Paths via Correlation Matrix Rank [71.09032766271493]
  Large language models (LLMs) are prone to errors and hallucinations.<n>How to check their outputs effectively and efficiently has become a critical problem in their applications.
  arXiv  Detail & Related papers  (2025-10-28T11:01:10Z)
• No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes [2.6550928535945872]
  We train linear probes to predict whether a model's forthcoming answer will be correct.<n>Across three open-source model families, projections on this "in-advance correctness direction" trained on generic trivia questions predict success in distribution.<n>For models responding "I don't know", doing so strongly correlates with the probe score, indicating that the same direction also captures confidence.
  arXiv  Detail & Related papers  (2025-09-12T18:09:55Z)
• Inducing Faithfulness in Structured Reasoning via Counterfactual Sensitivity [6.908972852063454]
  Large language models often generate a correct answer while relying on a flawed or irrelevant reasoning trace.<n>This paper introduces textbfCounterfactual Sensitivity Regularization (CSR), a novel training objective.<n>CSR improves faithfulness over standard fine-tuning and process supervision by up to 70 percentage points.
  arXiv  Detail & Related papers  (2025-09-01T15:18:46Z)
• SEAL: Steerable Reasoning Calibration of Large Language Models for Free [58.931194824519935]
  Large Language Models (LLMs) have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism.<n>Recent studies reveal substantial redundancy in the CoT reasoning traces, which negatively impacts model performance.<n>We introduce SEAL, a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains.
  arXiv  Detail & Related papers  (2025-04-07T02:42:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.