Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report

• URL: http://arxiv.org/abs/2604.17707v1
• Date: Mon, 20 Apr 2026 01:42:54 GMT
• Title: Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report
• Abstract summary: We apply the validity scaling framework from the PAI and MMPI-3 to metacognitive probe data.<n>Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring) and TRIN.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI and MMPI-3 to metacognitive probe data from 20 frontier models across 524 items. Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring), and TRIN (fixed responding). A tiered classification system identifies four models as construct-level invalid and two as elevated. Valid-profile models produce item-sensitive confidence (mean r = .18, 14 of 16 significant). Invalid-profile models do not (mean r = -.20, d = 2.17, p = .001). Chain-of-thought training produces two opposite response distortions. Two latent dimensions account for 94.6% of index variance. Companion papers extract a portable screening protocol (Cacioli, 2026e) and validate it against selective prediction (Cacioli, 2026f). All data and code: https://github.com/synthiumjp/validity-scaling-llm

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis [42.62413276719373]
  A trustworthy result should be supported by a valid path from the user question to the relevant data evidence.<n>This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms.<n>We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness.
  arXiv  Detail & Related papers  (2026-08-25T07:09:01Z)
• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• The Metacognitive Probe: Five Behavioural Calibration Diagnostics for LLMs [0.0]
  The Metacognitive Probe is an exploratory five-task, 15-slot diagnostic.<n>It decomposes an LLM's confidence behaviour into five behaviourally-distinct dimensions.<n>It is evaluated on N=69 humans.
  arXiv  Detail & Related papers  (2026-05-11T00:55:31Z)
• Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen [0.0]
  We tested whether seven instruction-tuned open-weight models produce verbalised confidence that meets minimal validity criteria.<n>We found Categorical elicitation did not rescue validity.<n> Token-level logprobability did not usefully predict verbalised confidence under the observed variance regime.
  arXiv  Detail & Related papers  (2026-04-24T04:45:21Z)
• Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction [0.0]
  The validity screen classifies LLM confidence signals as Valid, Indeterminate, or Invalid.<n>Twenty frontier LLMs from seven families were evaluated on 524 items across six cognitive tracks.<n>Split-half cross-validation yields median d = 1.77, P(d > 0) = 1.0 across 1,000 splits.
  arXiv  Detail & Related papers  (2026-04-20T01:56:29Z)
• Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals [0.0]
  LLM confidence signals are used for abstention, routing, and safety-critical decisions.<n>No standard practice exists for checking whether a confidence signal carries item-level information before building on it.<n>We transfer the validity screening principle from clinical personality assessment (PAI, MMPI-3) as a portable protocol for benchmark-based LLM confidence data.
  arXiv  Detail & Related papers  (2026-04-20T01:50:38Z)
• The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring [0.0]
  We introduce a cross-domain behavioural assay of monitoring-control coupling in LLMs, grounded in the Nelson and Narens (1990) metacognitive framework.<n>The battery-dependent calibration comprises monotonically decreasing (Q), monotonically increasing (GPT-5.4), or flat (Gemma)<n>Findings converge structurally with an independent Type-2 SDT approach, providing preliminary cross-method calibration.
  arXiv  Detail & Related papers  (2026-04-17T05:15:15Z)
• How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles [46.63622714488747]
  Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, latent entanglement.<n>In practice, this manifests as correlated reasoning patterns and synchronized failures.<n>We develop a statistical framework for auditing behavioral entanglement among black-box language models.
  arXiv  Detail & Related papers  (2026-04-08T23:32:06Z)
• NANOZK: Layerwise Zero-Knowledge Proofs for Verifiable Large Language Model Inference [0.0]
  We present METHOD, a zero-knowledge proof system that makes LLM inference verifiable.<n>Our approach exploits the fact that transformer inference naturally decomposes into independent layer computations.<n>Compared to EZKL, METHOD achieves 70x smaller proofs and 5.7x faster proving time at d=128, while maintaining formal soundness guarantees.
  arXiv  Detail & Related papers  (2026-03-17T04:14:45Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Dissecting Human and LLM Preferences [80.55271307662365]
  We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. We show that preference-based evaluation can be intentionally manipulated.
  arXiv  Detail & Related papers  (2024-02-17T14:34:31Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)
• ADT-SSL: Adaptive Dual-Threshold for Semi-Supervised Learning [68.53717108812297]
  Semi-Supervised Learning (SSL) has advanced classification tasks by inputting both labeled and unlabeled data to train a model jointly. This paper proposes an Adaptive Dual-Threshold method for Semi-Supervised Learning (ADT-SSL) Experimental results show that the proposed ADT-SSL achieves state-of-the-art classification accuracy.
  arXiv  Detail & Related papers  (2022-05-21T11:52:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.