Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

• URL: http://arxiv.org/abs/2605.26045v1
• Date: Mon, 25 May 2026 17:08:47 GMT
• Title: Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
• Abstract summary: We investigate 6 different methods for estimating the confidence of activation oracles.<n> bootstrap mode frequency is the best-calibrated method among those tested.<n>The log-prob baseline can serve as a fast triage signal at a fraction of the cost.
• Score: 2.4305775926851334
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Activation oracles aim to make the activations of other models legible to humans and yield promising results compared to white-box interpretability techniques. However, uncertainty quantification (UQ) for the natural-language outputs of such activation oracles is so far understudied. Here, we investigate 6 different methods for estimating the confidence of activation oracles and evaluate how well-calibrated their confidence scores are. Our experiments on 6,000 samples per oracle (varying verbalizer and context prompts) reveal that bootstrap mode frequency is the best-calibrated method among those tested (ECE 5.7% vs. 25.5% for the answer-word log-probability on Qwen3-8B; 10.3% vs. 13.1% on Qwen3.6-27B), and that the log-prob baseline can serve as a fast triage signal at a fraction of the cost. Code and the patched trainer are available at https://github.com/federicotorrielli/probabilistic_activation_oracles.

Related papers

• Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem [14.44743150931186]
  invalid or underspecified inputs can create spurious fault detections and apparent evolutionary gains.<n>We audit this failure mode in feedback-driven test generation using 142 development tasks, 114 locked external tasks, and 138 held-out tasks.<n>We propose an audit-and-placebo protocol that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators.
  arXiv  Detail & Related papers  (2026-08-20T04:29:35Z)
• A Controlled Study of Decoding-Time Truthfulness Methods on Instruction-Tuned LLMs [3.4007995136788]
  Layer-contrast decoding, inference-time intervention, and learned logit adapters have demonstrated 10-30 point gains on TruthfulQA.<n>Modern instruction-tuned LLMs already achieve substantially higher baselines.<n> Deliberative prompting methods appear more robust in the evaluated regime.
  arXiv  Detail & Related papers  (2026-06-10T14:48:05Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• A Paired Testing Protocol for Batch-Conditioned Refusal Robustness in LLM Serving [0.0]
  Safety evaluations of language models often treat serving configuration as fixed background infrastructure.<n>We synthesize four artifact-backed studies into a paired testing protocol.<n>Standard vLLM reproduces 22/55 label flips on current score-flip candidates, while enabling VLLM_BATCH_INIANT=1 reduces the same test to 0/55 flips.
  arXiv  Detail & Related papers  (2026-05-26T23:22:55Z)
• From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models [0.0]
  We propose a five-stage methodology for causal feature analysis in transformer language models.<n>We demonstrate it end-to-end on GPT-2 small performing the Indirect Object Identification (IOI) task.
  arXiv  Detail & Related papers  (2026-05-21T13:25:16Z)
• What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models [0.0]
  Single-prompt accuracy is the dominant way to benchmark language models, but it can miss reliability failures that matter.<n>We evaluate a 15-model open-weight corpus, with the main reliability analyses focused on 10 instruct models across five classification and reasoning benchmarks.<n>We find three broad results. First, evaluation design can materially change the conclusion.<n>Second, confidence signals are fragile. On MMLU-Pro, every primary model verbally reports confidence substantially above both its accuracy and its token-probability confidence on the same rows, and verbal parse rate can collapse for a single model on a single prompt variant.
  arXiv  Detail & Related papers  (2026-05-03T20:05:08Z)
• Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen [0.0]
  We tested whether seven instruction-tuned open-weight models produce verbalised confidence that meets minimal validity criteria.<n>We found Categorical elicitation did not rescue validity.<n> Token-level logprobability did not usefully predict verbalised confidence under the observed variance regime.
  arXiv  Detail & Related papers  (2026-04-24T04:45:21Z)
• Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models? [0.0]
  Chain-of-thought (CoT) reasoning has been proposed as a transparency mechanism for large language models in safety-critical deployments.<n>This study tests 12 open-weight reasoning models spanning 9 architectural families on 498 multiple-choice questions.<n>Across 41,832 inference runs, overall faithfulness rates range from 39.7% (Seed-1.6-Flash) to 89.9% (DeepSeek-V3.2-Speciale) across model families.
  arXiv  Detail & Related papers  (2026-03-23T21:21:37Z)
• Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation [0.0]
  Recent work on chain-of-thought faithfulness reports single aggregate numbers.<n>This paper provides evidence that faithfulness is not an objective, measurable property of a model.
  arXiv  Detail & Related papers  (2026-03-20T17:48:43Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs [70.4578433679737]
  We introduce Audio-Visual Trustworthiness assessment Benchmark (AVTrustBench), comprising 600K samples spanning over 9 meticulously crafted tasks.<n>Using our benchmark we extensively evaluate 13 state-of-the-art AVLLMs.<n>The findings reveal that the majority of existing models fall significantly short of achieving human-like comprehension.
  arXiv  Detail & Related papers  (2025-01-03T23:03:24Z)
• Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation [70.27452774899189]
  Large language models (LLMs) should signal low confidence on examples where they are incorrect, instead of misleading the user. As of November 2023, state-of-the-art LLMs do not provide access to these probabilities. Our best method composing linguistic confidences and surrogate model probabilities gives state-of-the-art confidence estimates on all 12 datasets.
  arXiv  Detail & Related papers  (2023-11-15T11:27:44Z)
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units [81.53783563025084]
  We propose an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only. HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.
  arXiv  Detail & Related papers  (2021-06-14T14:14:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.