Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas

• URL: http://arxiv.org/abs/2605.06673v1
• Date: Tue, 21 Apr 2026 12:44:47 GMT
• Title: Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas
• Abstract summary: Applied/Professional knowledge was the easiest benchmark domain to monitor.<n>Formal Reasoning and Natural Science were reliably the hardest.<n>Within-family profile-shape clustering is significant for Anthropic, Google-Gemini, and Qwen.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Aggregate metacognitive quality scores mask within-model variation across MMLU benchmark domains. We administered 1,500 MMLU items (250 per domain, under an a priori six-domain grouping) to 33 frontier LLMs from eight model families and computed Type-2 AUROC per model-domain cell using verbalized confidence (0-100). Total observations: 47,151. Every model with above-chance aggregate monitoring showed non-trivial domain-level variation. Applied/Professional knowledge was reliably the easiest benchmark domain to monitor (mean AUROC = .742, ranked top-2 in 21 of 33 models); Formal Reasoning and Natural Science were reliably the hardest (one of the two ranked bottom-2 in 27 of 33 models). The three middle domains were statistically indistinguishable (Kendall's W = .164). A subject-level coherence analysis (within-domain similarity ratio = 0.95) confirms the six-domain grouping is a pragmatic benchmark taxonomy, not a validated latent construct. Within-family profile-shape clustering is significant for Anthropic, Google-Gemini, and Qwen (permutation p < .0001) but not DeepSeek, Google-Gemma, or OpenAI. Gemma 4 31B showed a +.202 AUROC improvement over Gemma 3 27B. Three models classified Invalid on binary KEEP/WITHDRAW probes produced normal profiles under verbalized confidence, confirming probe-format specificity. Bootstrap 95% CIs on 198 cells have median width .199. Split-half aggregate stability r = .893; profile-level split-half is weaker (grand median r = .184). These results show stable benchmark-domain variation obscured by aggregate metrics, and support benchmark-stage domain screening as a step before deployment in specific application areas.

Related papers

• MED-DSLC: Multi-Expert-Domain Classification via Domain Supervision and Logit Calibration [42.68764270775848]
  Vision-language models (VLMs) enable zero-shot classification by comparing image features with text prompts in a shared embedding space.<n>VLMs are often adapted to fine-grained domains using techniques such as LoRA.<n>This leads to a highly fragmented model ecosystem, with thousands of specialized models.
  arXiv  Detail & Related papers  (2026-07-13T01:18:32Z)
• How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings? [58.740078141879984]
  Machine learning models often degrade when they are deployed on a target distribution that differs from the source distributions they were trained on.<n>Recent work in causality-based domain generalization has shown how shared causal structure can induce invariant predictors.<n>In this paper, we study when full or partial causal knowledge can provably improve supervised domain adaptation.
  arXiv  Detail & Related papers  (2026-06-10T21:07:49Z)
• The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive [0.0]
  CPU-only scoring primitive runs at 2.6 microseconds per token.<n> token rank-frequency distributions converge to the same two- parameter Mandelbrot ranking distribution.<n>We derive a single-pass scoring primitive that composes with model log probabilities when available and degrades to a rank-only mode usable on closed APIs.
  arXiv  Detail & Related papers  (2026-04-28T13:35:31Z)
• Quantisation Reshapes the Metacognitive Geometry of Language Models [0.0]
  We evaluate Llama-3-8B-Instruct on the same 3,000 questions at Q5_K_M and f16 precision.<n>We find that M-ratio profiles across four knowledge domains are uncorrelated between formats.
  arXiv  Detail & Related papers  (2026-04-10T05:29:22Z)
• Granular Ball Guided Stable Latent Domain Discovery for Domain-General Crowd Counting [19.18297173252027]
  We propose a granular ball guided stable latent domain discovery framework for domain-general crowd counting.<n>The proposed method first groups samples into compact local granular balls and then clusters granular ball centers as representatives to infer pseudo-domains.<n>On top of the discovered latent domains, we develop a two-branch learning framework that improves transferable semantic representations.
  arXiv  Detail & Related papers  (2026-03-25T09:12:35Z)
• ADNet: A Large-Scale and Extensible Multi-Domain Benchmark for Anomaly Detection Across 380 Real-World Categories [26.951550574484553]
  We introduce ADNet, a large-scale, multi-domain benchmark for anomaly detection.<n>The benchmark includes a total of 196,294 RGB images, consisting of 116,192 normal samples for training and 80,102 test images, of which 60,311 are anomalous.<n>Dinomaly-m is a context-guided Mixture-of-Experts that expands decoder capacity without increasing inference cost.
  arXiv  Detail & Related papers  (2025-11-25T10:47:48Z)
• Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores [3.959606869996232]
  We show that averaging TVD-MI's binary trials yields centered-probability scores with additive structure suitable for item-response theory (IRT) without nonlinear link functions.<n>We derive this clipped-linear evaluations from Gini entropy, yielding a box-constrained least-squares formulation that handles boundary saturation.
  arXiv  Detail & Related papers  (2025-10-16T17:59:25Z)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling [91.76069784586149]
  Prior work generally assumes that process reward models (PRMs) outperform outcome reward models (ORMs) that assess only the final answer.<n>We present the first unified evaluation of four reward model variants across 14 diverse domains.<n>We attribute this to PRM-style stepwise scoring, which inherits label noise from LLM auto-labeling and has difficulty evaluating long reasoning trajectories.
  arXiv  Detail & Related papers  (2025-10-01T04:21:14Z)
• Multi-Prompt Progressive Alignment for Multi-Source Unsupervised Domain Adaptation [73.40696661117408]
  We propose a progressive alignment strategy for adapting CLIP to unlabeled downstream task.<n>We name our approach MP2A and test it on three popular UDA benchmarks, namely ImageCLEF, Office-Home, and the most challenging DomainNet.<n> Experiments showcase that MP2A achieves state-of-the-art performance when compared with most recent CLIP-based MS-UDA approaches.
  arXiv  Detail & Related papers  (2025-07-31T09:42:42Z)
• Bidirectional Domain Mixup for Domain Adaptive Semantic Segmentation [73.3083304858763]
  This paper systematically studies the impact of mixup under the domain adaptaive semantic segmentation task. In specific, we achieve domain mixup in two-step: cut and paste. We provide extensive ablation experiments to empirically verify our main components of the framework.
  arXiv  Detail & Related papers  (2023-03-17T05:22:44Z)
• Improving Domain Generalization with Domain Relations [77.63345406973097]
  This paper focuses on domain shifts, which occur when the model is applied to new domains that are different from the ones it was trained on. We propose a new approach called D$3$G to learn domain-specific models. Our results show that D$3$G consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-02-06T08:11:16Z)
• Low-confidence Samples Matter for Domain Adaptation [47.552605279925736]
  Domain adaptation (DA) aims to transfer knowledge from a label-rich source domain to a related but label-scarce target domain. We propose a novel contrastive learning method by processing low-confidence samples. We evaluate the proposed method in both unsupervised and semi-supervised DA settings.
  arXiv  Detail & Related papers  (2022-02-06T15:45:45Z)
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization [63.28279815753543]
  In Domain Generalization (DG) settings, models trained on a given set of training domains have notoriously chaotic performance on shifted test domains. We first show that a simple protocol for averaging model parameters along the optimization path, starting early during training, significantly boosts domain generalizationity. We show that an ensemble of independently trained models also has a chaotic behavior in the DG setting.
  arXiv  Detail & Related papers  (2021-10-21T00:08:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.