Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making

• URL: http://arxiv.org/abs/2607.14277v1
• Date: Wed, 15 Jul 2026 18:36:54 GMT
• Title: Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making
• Abstract summary: We introduce Multi-Head Latent Control, a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to produce deployment-time control signals.<n>A Capability Head predicts whether the current model can solve the instance or should defer to a stronger collaborator, while a Resolution Head predicts appropriate resolution decision Clarification, Tool Use, Abstention, or Direct Answering.<n>Both heads are trained only on latent traces from the same frozen LLM backbone, enabling post hoc adaptation without modifying the model.
• Score: 15.37854161578634
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly deployed as agents, but reliable agentic behavior requires more than next-token prediction. At inference time, it is preferred that an agent can decide whether to proceed with its current reasoning, defer to a stronger model, request additional information, invoke external tools, or abstain under the given setup. Existing approaches address these decisions through prompt-level routing, external orchestration, or task-specific fine-tuning, which primarily rely on input-side signals, and are often costly and difficult to maintain as model backbones evolve. We ask whether such control decisions can be inferred directly from a model's latent generation process. We introduce Multi-Head Latent Control, a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to produce deployment-time control signals. A Capability Head predicts whether the current model can solve the instance or should defer to a stronger collaborator, while a Resolution Head predicts appropriate resolution decision Clarification, Tool Use, Abstention, or Direct Answering. Both heads are trained only on latent traces from the same frozen LLM backbone, enabling post hoc adaptation without modifying the model. Across language and vision-language settings, Multi-Head Latent Control consistently improves the quality-cost tradeoff of multi-model systems, enabling early handoff from partial generations and more accurate intervention decisions. In routed execution (small + large model), it reduces large-model usage by up to 90.7 percent on AndroidWorld and 27-53 percent on average across benchmarks, while retaining most of large-model performance. Additionally, the learned control signals improve tool-use decision quality, yielding up to +158 percent relative score gain and 65.5 percent fewer missed-required tool calls.

Related papers

• Long-Horizon Embodied Decision-Making via Multimodal Memory Compression [71.16335993005139]
  DunphyBench is a new benchmark for evaluating agents on long-horizon human-centered embodied decision-making.<n>MeMento is a preference-conditioned multimodal memory compressor that selectively compresses decision-relevant information based on user preferences.<n>MeMento helps VLM-driven agents improve accuracy by 7.18%, while reducing memory usage by 85.38% compared to the strongest baseline.
  arXiv  Detail & Related papers  (2026-08-02T19:29:48Z)
• Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning [87.18590662029432]
  We propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models.<n>Our draft Rating Learning (DRL) equips the draft model with an internal confidence estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query.<n>These ratings enable fine-grained, instance-level textbfProactive Routing and substantially accelerate inference without compromising overall performance.
  arXiv  Detail & Related papers  (2026-06-29T12:30:24Z)
• Reliability and Effectiveness of Autonomous AI Agents in Supply Chain Management [15.19525853427301]
  This paper studies autonomous generative AI agents in multi-echelon supply chains using the MIT Beer Game.<n>We identify four inference-time levers that shape performance: model selection, policies and guardrails, centralized data sharing, and prompt engineering.
  arXiv  Detail & Related papers  (2026-05-16T15:11:35Z)
• Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use [47.29360932085394]
  Large language models (LLMs) increasingly act as autonomous agents that must decide when to answer directly vs. when to invoke external tools.<n>We introduce a model-adaptive definition of tool-necessity, grounded in each model's empirical performance.<n>We find substantial mismatches of 26.5-54.0% and 30.8-41.8%, respectively.
  arXiv  Detail & Related papers  (2026-05-13T18:59:28Z)
• Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs [31.34021188099265]
  Multi-agent large language model (LLM) systems often rely on a controller to coordinate a pool of heterogeneous models.<n>We propose a critique-and-routing controller that casts multi-agent coordination as a sequential decision problem.
  arXiv  Detail & Related papers  (2026-05-09T04:51:42Z)
• Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference [10.009730627424629]
  Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks.<n>We propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates.
  arXiv  Detail & Related papers  (2026-02-25T16:38:03Z)
• Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads [104.9566359759396]
  We propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty scores.<n>Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification.
  arXiv  Detail & Related papers  (2025-11-09T03:38:29Z)
• Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection [71.8243083897721]
  Vision-language models often hallucinate details, generating non-existent objects or inaccurate attributes that compromise output reliability.<n>We present a novel framework that leverages the model's self-consistency between long responses and short answers to generate preference pairs for training.
  arXiv  Detail & Related papers  (2025-09-27T10:37:11Z)
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model [119.65409513119963]
  We introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods.
  arXiv  Detail & Related papers  (2023-05-29T17:57:46Z)
• Predictable MDP Abstraction for Unsupervised Model-Based RL [93.91375268580806]
  We propose predictable MDP abstraction (PMA) Instead of training a predictive model on the original MDP, we train a model on a transformed MDP with a learned action space. We theoretically analyze PMA and empirically demonstrate that PMA leads to significant improvements over prior unsupervised model-based RL approaches.
  arXiv  Detail & Related papers  (2023-02-08T07:37:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.