Metacognitive Sensitivity for Test-Time Dynamic Model Selection

• URL: http://arxiv.org/abs/2512.10451v1
• Date: Thu, 11 Dec 2025 09:15:05 GMT
• Title: Metacognitive Sensitivity for Test-Time Dynamic Model Selection
• Authors: Le Tuan Minh Trinh, Le Minh Vu Pham, Thi Minh Anh Pham, An Duc Nguyen,
• Abstract summary: We propose a new framework for evaluating and leveraging AI metacognition.<n>We introduce meta-d', a psychologically-grounded measure of metacognitive sensitivity, to characterise how reliably a model's confidence predicts its own accuracy.<n>We then use this dynamic sensitivity score as context for a bandit-based arbiter that performs test-time model selection.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A key aspect of human cognition is metacognition - the ability to assess one's own knowledge and judgment reliability. While deep learning models can express confidence in their predictions, they often suffer from poor calibration, a cognitive bias where expressed confidence does not reflect true competence. Do models truly know what they know? Drawing from human cognitive science, we propose a new framework for evaluating and leveraging AI metacognition. We introduce meta-d', a psychologically-grounded measure of metacognitive sensitivity, to characterise how reliably a model's confidence predicts its own accuracy. We then use this dynamic sensitivity score as context for a bandit-based arbiter that performs test-time model selection, learning which of several expert models to trust for a given task. Our experiments across multiple datasets and deep learning model combinations (including CNNs and VLMs) demonstrate that this metacognitive approach improves joint-inference accuracy over constituent models. This work provides a novel behavioural account of AI models, recasting ensemble selection as a problem of evaluating both short-term signals (confidence prediction scores) and medium-term traits (metacognitive sensitivity).

Related papers

• Evaluating Human and Machine Confidence in Phishing Email Detection: A Comparative Study [0.45961260934000997]
  This research examines how human cognition and machine learn- ing models work together to distinguish phishing emails from legitimate ones.<n>Our results show that machine learning models provide good accuracy rates, but their confidence levels vary significantly.<n>Human evaluators, on the other hand, use a greater variety of language signs and retain more consistent confidence.
  arXiv  Detail & Related papers  (2026-01-08T05:30:41Z)
• BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents [58.05949210993854]
  We investigate whether search agents have the ability to communicate their own confidence through verbalized confidence scores after long sequences of actions.<n>We propose Test-Time Scaling (TTS) methods that use confidence scores to determine answer quality, encourage the model to try again until reaching a satisfactory confidence level.
  arXiv  Detail & Related papers  (2025-10-27T15:58:51Z)
• Know Thyself? On the Incapability and Implications of AI Self-Recognition [22.582593406983907]
  Self-recognition is a crucial metacognitive capability for AI systems, relevant not only for psychological analysis but also for safety.<n>We introduce a systematic evaluation framework that can be easily applied and updated.<n>We measure how well 10 contemporary larger language models (LLMs) can identify their own generated text versus text from other models.
  arXiv  Detail & Related papers  (2025-10-03T18:00:01Z)
• Uncertainty Estimation by Human Perception versus Neural Models [7.273936646577467]
  Modern neural networks (NNs) often achieve high predictive accuracy but are poorly calibrated.<n>We investigate how human uncertainty compares to uncertainty estimated by NNs.
  arXiv  Detail & Related papers  (2025-06-18T20:00:20Z)
• Truly Self-Improving Agents Require Intrinsic Metacognitive Learning [59.60803539959191]
  Self-improving agents aim to continuously acquire new capabilities with minimal supervision.<n>Current approaches face two key limitations: their self-improvement processes are often rigid, fail to generalize across tasks domains, and struggle to scale with increasing agent capabilities.<n>We argue that effective self-improvement requires intrinsic metacognitive learning, defined as an agent's intrinsic ability to actively evaluate, reflect on, and adapt its own learning processes.
  arXiv  Detail & Related papers  (2025-06-05T14:53:35Z)
• Dynamic Programming Techniques for Enhancing Cognitive Representation in Knowledge Tracing [125.75923987618977]
  We propose the Cognitive Representation Dynamic Programming based Knowledge Tracing (CRDP-KT) model.<n>It is a dynamic programming algorithm to optimize cognitive representations based on the difficulty of the questions and the performance intervals between them.<n>It provides more accurate and systematic input features for subsequent model training, thereby minimizing distortion in the simulation of cognitive states.
  arXiv  Detail & Related papers  (2025-06-03T14:44:48Z)
• On the Fairness, Diversity and Reliability of Text-to-Image Generative Models [68.62012304574012]
  multimodal generative models have sparked critical discussions on their reliability, fairness and potential for misuse.<n>We propose an evaluation framework to assess model reliability by analyzing responses to global and local perturbations in the embedding space.<n>Our method lays the groundwork for detecting unreliable, bias-injected models and tracing the provenance of embedded biases.
  arXiv  Detail & Related papers  (2024-11-21T09:46:55Z)
• Conceptual Belief-Informed Reinforcement Learning [10.817700298999]
  Reinforcement learning (RL) has achieved significant success but is hindered by inefficiency and instability.<n>We introduce Conceptual Belief-Informed Reinforcement Learning to emulate human intelligence (HI-RL)<n>HI-RL forms concepts by extracting high-level categories of critical environmental information and then constructs adaptive concept-associated probabilistic beliefs as experience priors to guide value or policy updates.
  arXiv  Detail & Related papers  (2024-10-02T16:50:29Z)
• Benchmarking Continual Learning from Cognitive Perspectives [14.867136605254975]
  Continual learning addresses the problem of continuously acquiring and transferring knowledge without catastrophic forgetting of old concepts. There is a mismatch between cognitive properties and evaluation methods of continual learning models. We propose to integrate model cognitive capacities and evaluation metrics into a unified evaluation paradigm.
  arXiv  Detail & Related papers  (2023-12-06T06:27:27Z)
• Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models [47.890846082224066]
  This work fills the gap by presenting CovidET-Appraisals, the most comprehensive dataset to-date that assesses 24 appraisal dimensions. CovidET-Appraisals presents an ideal testbed to evaluate the ability of large language models to automatically assess and explain cognitive appraisals.
  arXiv  Detail & Related papers  (2023-10-22T19:12:17Z)
• Backprop-Free Reinforcement Learning with Active Neural Generative Coding [84.11376568625353]
  We propose a computational framework for learning action-driven generative models without backpropagation of errors (backprop) in dynamic environments. We develop an intelligent agent that operates even with sparse rewards, drawing inspiration from the cognitive theory of planning as inference. The robust performance of our agent offers promising evidence that a backprop-free approach for neural inference and learning can drive goal-directed behavior.
  arXiv  Detail & Related papers  (2021-07-10T19:02:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.