TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints

• URL: http://arxiv.org/abs/2605.13414v1
• Date: Wed, 13 May 2026 12:10:05 GMT
• Title: TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints
• Abstract summary: We introduce TRIAGE, a framework for testing language models for metacognitive control.<n>We find that current language models exhibit substantial gaps in prospective metacognitive control.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to attempt, in what order, and how much compute to commit to each, all before any execution feedback is available. This is the prospective form of metacognitive control studied for decades in human cognition, yet whether language models possess it remains untested. We introduce TRIAGE, an evaluation framework in which a model receives a task pool and a token budget calibrated to its own baseline cost, and commits to a single ordered plan that jointly encodes selection, sequencing, and per-problem allocation. Plans are scored against an oracle with full knowledge of the model's solvability and cost on each problem, yielding a triage efficiency ratio on a common scale. We evaluate frontier and open-source models, with and without reasoning enabled, across competition mathematics, graduate-level science, code generation, and expert multidisciplinary knowledge, and find that current language models exhibit substantial gaps in prospective metacognitive control, revealing a previously unmeasured capability dimension with direct implications for resource-efficient agent deployment.

Related papers

• Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions [59.21329728895449]
  We introduce an exam-style evaluation framework for studying this setting.<n>We find that models fail to allocate a shared budget strategically across questions of varying difficulties and values.<n>These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation.
  arXiv  Detail & Related papers  (2026-08-08T07:04:28Z)
• RIDGE: An Autonomous Framework for Validation and Method Discovery in LLM-Generated Option Pricing [0.00707044523927933]
  RIDGE is an autonomous validation framework for generated pricing implementations.<n> Validation evidence is interpreted diagnostically, while the resulting knowledge is accumulated in a repository and reused across models.<n>All detected implementation defects are removed and, in two cases, the validation process reveals methodological limitations and motivates the development of alternative numerical methods.
  arXiv  Detail & Related papers  (2026-07-28T02:12:52Z)
• Adaptive Multi-Round Allocation with Stochastic Arrivals [26.102812388131813]
  We study a sequential resource allocation problem motivated by adaptive network recruitment.<n>We first show that the single-round allocation problem admits an exact greedy solution based on marginal survival probabilities.<n>In the multi-round setting, the resulting Bellman recursion is intractable due to the high-dimensional evolution of the frontier.
  arXiv  Detail & Related papers  (2026-05-12T13:29:06Z)
• Reasoning Is All You Need for Urban Planning AI [3.3943213418026126]
  This paper presents the Agentic Urban Planning AI Framework for reasoning-capable planning agents.<n>It integrates three cognitive layers (Perception, Foundation, Reasoning) with six logic components (Analysis, Generation, Verification, Evaluation, Collaboration, Decision) through a multi-agents collaboration framework.<n>We show how AI agents can augment human planners by systematically exploring solution spaces, verifying regulatory compliance, and deliberating over trade-offs transparently.
  arXiv  Detail & Related papers  (2025-11-07T15:59:06Z)
• Enhancing Credit Risk Prediction: A Meta-Learning Framework Integrating Baseline Models, LASSO, and ECOC for Superior Accuracy [7.254744067646655]
  This research proposes a comprehensive meta-learning framework that synthesizes multiple complementary models.<n>We implement Permutation Feature Importance analysis for each prediction class across all constituent models.<n>Results demonstrate that our framework significantly enhances the accuracy of financial entity classification.
  arXiv  Detail & Related papers  (2025-09-26T14:09:04Z)
• Hell or High Water: Evaluating Agentic Recovery from External Failures [19.402114652658206]
  We develop a specialized agentic planning benchmark to study how well language agents search for alternative ways to achieve their goals.<n>Our benchmark confronts the agent with external failures in its workflow, such as functions that suddenly become unavailable.<n>Overall, we find that language agents struggle to formulate and execute backup plans in response to environment feedback.
  arXiv  Detail & Related papers  (2025-08-14T19:21:09Z)
• Cascaded Language Models for Cost-effective Human-AI Decision-Making [52.81324217423194]
  We present a cascaded LLM decision framework that adaptively delegates tasks across multiple tiers of expertise.<n>First, a deferral policy determines whether to accept the base model's answer or regenerate it with a large model.<n>Second, an abstention policy decides whether the cascade model response is sufficiently certain or requires human intervention.
  arXiv  Detail & Related papers  (2025-06-13T15:36:22Z)
• Agentic Knowledgeable Self-awareness [79.25908923383776]
  KnowSelf is a data-centric approach that applies agents with knowledgeable self-awareness like humans.<n>Our experiments demonstrate that KnowSelf can outperform various strong baselines on different tasks and models with minimal use of external knowledge.
  arXiv  Detail & Related papers  (2025-04-04T16:03:38Z)
• Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planning [17.752461521448236]
  This paper focuses on decision planning with uncertainty estimation to address the problem in language models. Our uncertainty estimation and decision-making agent design offer a cost-efficient approach for AI agent development.
  arXiv  Detail & Related papers  (2024-02-01T00:23:31Z)
• Modeling Boundedly Rational Agents with Latent Inference Budgets [56.24971011281947]
  We introduce a latent inference budget model (L-IBM) that models agents' computational constraints explicitly. L-IBMs make it possible to learn agent models using data from diverse populations of suboptimal actors. We show that L-IBMs match or outperform Boltzmann models of decision-making under uncertainty.
  arXiv  Detail & Related papers  (2023-12-07T03:55:51Z)
• On Exploiting Hitting Sets for Model Reconciliation [53.81101846598925]
  In human-aware planning, a planning agent may need to provide an explanation to a human user on why its plan is optimal. A popular approach to do this is called model reconciliation, where the agent tries to reconcile the differences in its model and the human's model. We present a logic-based framework for model reconciliation that extends beyond the realm of planning.
  arXiv  Detail & Related papers  (2020-12-16T21:25:53Z)
• Coordinated Online Learning for Multi-Agent Systems with Coupled Constraints and Perturbed Utility Observations [91.02019381927236]
  We introduce a novel method to steer the agents toward a stable population state, fulfilling the given resource constraints. The proposed method is a decentralized resource pricing method based on the resource loads resulting from the augmentation of the game's Lagrangian.
  arXiv  Detail & Related papers  (2020-10-21T10:11:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.