MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models

• URL: http://arxiv.org/abs/2604.19809v1
• Date: Wed, 15 Apr 2026 08:41:12 GMT
• Title: MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models
• Abstract summary: MIRROR is a benchmark that evaluates whether large language models can use self-knowledge to make better decisions.<n>We evaluate 16 models from 8 labs across approximately 250,000 evaluation instances.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce MIRROR, a benchmark comprising eight experiments across four metacognitive levels that evaluates whether large language models can use self-knowledge to make better decisions. We evaluate 16 models from 8 labs across approximately 250,000 evaluation instances using five independent behavioral measurement channels. Core experiments are run across the full model roster; experiments with specialized infrastructure requirements report explicitly marked model subsets. We find two phenomena with direct implications for agentic deployment: (1) compositional self-prediction fails universally -- the Compositional Calibration Error ranges from 0.500 to 0.943 on the original 15-model Exp3-v1 set (and 0.434 to 0.758 on the balanced 16-model Exp3-v2 expansion), indicating that models cannot predict their own performance on multi-domain tasks, and (2) models exhibit above-chance but imperfect domain-specific self-knowledge yet systematically fail to translate even this partial awareness into appropriate agentic action-selection -- external metacognitive control reduces the Confident Failure Rate from 0.600 to 0.143 (76% reduction at temperature 0; mean 70% at temperature 0.7 across 5 models from 4 labs). Providing models with their own calibration scores produces no significant improvement (p > 0.05); only architectural constraint is effective. This suggests that external metacognitive scaffolding -- not improved self-knowledge -- is the path to safer autonomous AI systems. Code, data, and Croissant metadata will be released publicly with the benchmark.

Related papers

• Small Foundation Models of Human Cognition and Behaviour [45.88028371034407]
  We train 14 models from 135M to 14B parameters across four architecture families on Psych-101.<n>Psych-101 is a dataset of 10.7 million trial-level choices from 160 experiments.<n>We show that small cognitively fine-tuned models show promise as noise ceiling estimators for psychological experiments.
  arXiv  Detail & Related papers  (2026-08-05T11:34:20Z)
• Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models [0.0]
  We present a systematic zero-shot evaluation of 41 open-weight language models spanning 15 families and the 135M--9B parameter range.<n>We analyze confidence calibration, robustness to realistic input perturbations, statistical reliability of model rankings, deployment efficiency, and benchmark saturation.<n>These findings provide practical guidance for selecting and evaluating open-weight language models for intent classification.
  arXiv  Detail & Related papers  (2026-07-29T19:42:33Z)
• Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI [2.2616066835313253]
  We run 86 picoCTF challenges across seven categories and three difficulty tiers, under three tool-access regimes and three model/client configurations.<n>We apply corrections to existing tools, agent-behavior changes, and eleven new capability tools, and re-run the previously-unsuccessful trials.<n>The overall solve rate rises from 55.4% to 72.0%, and every configuration improves significantly.
  arXiv  Detail & Related papers  (2026-07-03T02:20:04Z)
• What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models [0.0]
  Single-prompt accuracy is the dominant way to benchmark language models, but it can miss reliability failures that matter.<n>We evaluate a 15-model open-weight corpus, with the main reliability analyses focused on 10 instruct models across five classification and reasoning benchmarks.<n>We find three broad results. First, evaluation design can materially change the conclusion.<n>Second, confidence signals are fragile. On MMLU-Pro, every primary model verbally reports confidence substantially above both its accuracy and its token-probability confidence on the same rows, and verbal parse rate can collapse for a single model on a single prompt variant.
  arXiv  Detail & Related papers  (2026-05-03T20:05:08Z)
• Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels [0.0]
  Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment.<n>Existing studies typically compare only two conditions, rely on aggregate bias metrics, and evaluate a single model family.<n>We conduct a controlled empirical study of three instruction-tuned models at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items.
  arXiv  Detail & Related papers  (2026-05-02T05:41:47Z)
• Open-Set Vein Biometric Recognition with Deep Metric Learning [0.0]
  We rigorously evaluate the computational boundaries of Deep Metric Learning (DML) under strict open-set constraints.<n>Our approach learns discnative L2-normalised embeddings and employs prototype-based matching with a similarity threshold.<n>On the large-scale MMCBNU 6000 benchmark, our best model (ResNet50-CBAM) achieves an OSCR of 0.9945, AUROC of 0.9974, and EER of 1.57%, maintaining high identification accuracy (99.6% Rank-1) while robustly rejecting unknown subjects.
  arXiv  Detail & Related papers  (2026-04-16T11:03:11Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles [46.63622714488747]
  Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, latent entanglement.<n>In practice, this manifests as correlated reasoning patterns and synchronized failures.<n>We develop a statistical framework for auditing behavioral entanglement among black-box language models.
  arXiv  Detail & Related papers  (2026-04-08T23:32:06Z)
• Brevity Constraints Reverse Performance Hierarchies in Language Models [0.0]
  On 7.7% of benchmark problems, larger language models underperform smaller ones by 28.4 percentage points despite 10-100x more parameters.<n>We identify the mechanism as spontaneous scale-dependent verbosity that introduces errors through overelaboration.<n>Our results establish that maximizing large model performance requires scale-aware prompt engineering.
  arXiv  Detail & Related papers  (2026-03-11T06:47:41Z)
• Decomposing LLM Self-Correction: The Accuracy-Correction Paradox and Error Depth Hypothesis [6.901585308625979]
  We decompose self-correction into three sub-capabilities: error detection, error localization, and error correction.<n>Our findings challenge linear assumptions about model capability and self-improvement.
  arXiv  Detail & Related papers  (2025-12-24T21:51:24Z)
• Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluation [0.0]
  This study evaluates four abliteration tools across sixteen instruction-tuned models.<n>Single-pass methods demonstrated superior capability preservation on the benchmarked subset.<n>The principal finding indicates that mathematical reasoning capabilities exhibit the highest sensitivity to abliteration interventions.
  arXiv  Detail & Related papers  (2025-12-15T18:48:42Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models [0.0]
  AA- Omniscience is a benchmark designed to measure factual recall and knowledge calibration across 6,000 questions.<n>The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall.<n>Results reveal persistent factuality and calibration weaknesses across frontier models.
  arXiv  Detail & Related papers  (2025-11-17T06:27:16Z)
• SpecEval: Evaluating Model Adherence to Behavior Specifications [63.13000010340958]
  We introduce an automated framework that audits models against their providers specifications.<n>Our central focus is on three way consistency between a provider specification, its model outputs, and its own models as judges.<n>We apply our framework to 16 models from six developers across more than 100 behavioral statements, finding systematic inconsistencies including compliance gaps of up to 20 percent across providers.
  arXiv  Detail & Related papers  (2025-09-02T16:18:40Z)
• Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation [70.27452774899189]
  Large language models (LLMs) should signal low confidence on examples where they are incorrect, instead of misleading the user. As of November 2023, state-of-the-art LLMs do not provide access to these probabilities. Our best method composing linguistic confidences and surrogate model probabilities gives state-of-the-art confidence estimates on all 12 datasets.
  arXiv  Detail & Related papers  (2023-11-15T11:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.