Training Language Models to Cooperate with Inference-Time Controllers

• URL: http://arxiv.org/abs/2607.23771v1
• Date: Sun, 26 Jul 2026 17:44:33 GMT
• Title: Training Language Models to Cooperate with Inference-Time Controllers
• Abstract summary: CALM (Controller-Aware Language Models) is a post-training framework that explicitly places controllers in the training loop.<n>We formulate controller-aware post-training as multi-task reinforcement learning over controller-induced interaction protocols.<n>We evaluate CALM on held-out controller compositions and broader controller shifts, showing that controller-aware post-training improves generalization across inference-time beyond single-controller optimization.
• Score: 25.4633535418324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) performance increasingly depends not only on the base model, but also on the inference-time controller used to organize reasoning. Existing post-training methods, however, typically optimize for a single fixed interaction pattern, despite real deployments relying on diverse controllers such as Chain-of-Thought, self-consistency, debate, planning, and verification pipelines. This creates a training--deployment mismatch and limits transfer to new workflows. We introduce CALM (Controller-Aware Language Models), a post-training framework that explicitly places controllers in the training loop. We formulate controller-aware post-training as multi-task reinforcement learning over controller-induced interaction protocols, where controllers are compositions of reusable local reasoning modules. This structure also induces a module-level decomposition of mixed-controller training under a turn-level GRPO objective, enabling a systematic study of controller and module-aware training strategies. We evaluate CALM on held-out controller compositions and broader controller shifts, showing that controller-aware post-training improves generalization across inference-time workflows beyond single-controller optimization.

Related papers

• Agentic Visual Generation: From Generative Models to Agentic Control [73.3572965008302]
  In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors.<n>Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity.<n>We organize the field according to what the controller can directly control in the generation process.
  arXiv  Detail & Related papers  (2026-09-06T17:59:24Z)
• Heuristic Learning for Active Flow Control Using Coding Agents [2.867517731896504]
  We use modern coding agents to search directly for explicit executable feedback laws.<n>We introduce a constrained-learning protocol in which an agent iteratively proposes, evaluates, and revises controller implementations while interacting exclusively through the public benchmark interface.<n>The discovered controllers match or outperform the best DRL policy in 10 of the 13 environments while remaining compact, interpretable, and directly inspectable.
  arXiv  Detail & Related papers  (2026-07-13T13:47:17Z)
• AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipes [0.0]
  Manager reads structured telemetry snapshots from an active run, audits a constrained action space, and returns validated updates to training parameters.<n>On TinyStories, the manager detects and corrects overfitting, achieving a validation loss 60% lower than the baseline.<n>In a robotic manipulation reinforcement-learning task, we use the same bounded decision interface in an episodic closed-loop setting, where manager updates are applied at evaluation or checkpoint boundaries.
  arXiv  Detail & Related papers  (2026-06-29T07:07:33Z)
• Hierarchical Prompt-Domain Control and Learning for Resource-Constrained Agentic Language Models [0.0]
  Large Language Models are increasingly deployed inside agentic systems.<n>We propose a hierarchical control-and-learning framework in which a compact model is first distilled to learn the required output schema.<n>We show improved reliability and cost-efficiency over non-hierarchical, distillation-only, and non-distilled baselines.
  arXiv  Detail & Related papers  (2026-05-26T21:23:30Z)
• Small Model as Master Orchestrator: Learning Unified Agent-Tool Orchestration with Parallel Subtask Decomposition [61.291733522717415]
  Agent-as-Tool is a unified parallel orchestration paradigm that mitigates both agents and tools into a standardized, learnable action space.<n>ParaManager decouples planning decisions from subtask solving, enabling state-aware parallel subtask decomposition, delegation, and asynchronous execution.<n>Experiments show that ParaManager achieves strong performance across multiple benchmarks and exhibits robust generalization under unseen model pools.
  arXiv  Detail & Related papers  (2026-04-18T14:41:27Z)
• Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning [61.380634253724594]
  Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL)<n>We show that it is possible to overcome this problem by acting and exploring within the internal representations of an autoregressive model.
  arXiv  Detail & Related papers  (2025-12-23T18:51:50Z)
• EasySteer: A Unified Framework for High-Performance and Extensible LLM Steering [55.56674028743782]
  Large language model (LLM) steering has emerged as a promising paradigm for controlling model behavior at inference time.<n>We present EasySteer, a unified framework for high-performance, LLM steering built on vLLM.
  arXiv  Detail & Related papers  (2025-09-29T17:59:07Z)
• Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models [56.93608812478369]
  We present L2R, a method that isolates the training of new PEFT modules to ensure their task specialization. L2R then learns to compose the learned modules by training a network of routers that leverages a small memory containing examples of previously seen tasks. Our results demonstrate that L2R provides an effective composition of PEFT modules, leading to improved generalization and performance compared to other methods.
  arXiv  Detail & Related papers  (2024-08-16T23:57:29Z)
• Skip Training for Multi-Agent Reinforcement Learning Controller for Industrial Wave Energy Converters [94.84709449845352]
  Recent Wave Energy Converters (WEC) are equipped with multiple legs and generators to maximize energy generation. Traditional controllers have shown limitations to capture complex wave patterns and the controllers must efficiently maximize the energy capture. This paper introduces a Multi-Agent Reinforcement Learning controller (MARL), which outperforms the traditionally used spring damper controller.
  arXiv  Detail & Related papers  (2022-09-13T00:20:31Z)
• Deep Reinforcement Learning with Embedded LQR Controllers [1.256413718364189]
  We introduce a method that integrates LQR control into the action set, allowing generalization and avoiding fixing the computed control in the replay memory. In all cases, we show that adding LQR control can improve performance, although the effect is more profound if it can be used to augment a discrete action set.
  arXiv  Detail & Related papers  (2021-01-18T17:28:48Z)
• Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion [95.1825179206694]
  We present a framework that synthesizes robust controllers for a quadruped robot. A high-level controller learns to choose from a set of primitives in response to changes in the environment. A low-level controller that utilizes an established control method to robustly execute the primitives.
  arXiv  Detail & Related papers  (2020-09-21T16:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.