Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution

• URL: http://arxiv.org/abs/2606.20014v1
• Date: Thu, 18 Jun 2026 09:47:06 GMT
• Title: Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
• Abstract summary: Reinforcement learning (RL) has achieved strong performance in sequential decision-making.<n>We propose a hierarchical architecture where a pretrained large language model (LLM) acts as a central strategic controller.<n>We evaluate this hybrid system in a competitive 2v2 King of the Hill environment against behavior tree (BT) and emphFlat'' RL.
• Score: 33.648826376348985
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) has achieved strong performance in sequential decision-making, yet scaling to complex multi-agent environments remains challenging due to sparse rewards, large state-action spaces, and the difficulty of learning coordinated strategies. We propose a hierarchical architecture where a pretrained large language model (LLM) acts as a centralized strategic controller that selects among specialized RL skill policies for a team of agents, while RL policies handle reactive low-level execution. We evaluate this hybrid system in a competitive 2v2 King of the Hill environment against behavior tree (BT) and \emph{``Flat''} RL (end-to-end training without skill decomposition) baselines. The LLM+RL system achieves task performance statistically equivalent to hand-crafted BT (46.4\% vs 51.5\% win rate, $p=0.103$) while both significantly outperform Flat RL trained without skill decomposition. A user study ($n=15$) reveals that 60\% of participants perceive LLM+RL agents as the most human-like ($p=0.027$), citing behavioral adaptability and tactical variability. These results demonstrate that pretrained LLM reasoning can effectively orchestrate pretrained RL skills, achieving competitive multi-agent coordination and superior perceived believability without manual rule engineering.

Related papers

• Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL [55.05743310621117]
  Co-RL is a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers.<n>We show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops.
  arXiv  Detail & Related papers  (2026-08-18T01:16:02Z)
• SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling [54.276306194000405]
  We propose SortedRL, an online length-aware scheduling strategy for scaling reinforcement learning.<n>SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates.<n>Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, show that SortedRL reduces RL training bubble ratios by over 50%.
  arXiv  Detail & Related papers  (2026-03-24T16:48:31Z)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models [73.10315509190623]
  Recent reinforcement learning techniques have yielded impressive reasoning improvements in language models.<n>It remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training.<n>We develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training.
  arXiv  Detail & Related papers  (2025-12-08T18:12:10Z)
• Stronger Together: On-Policy Reinforcement Learning for Collaborative LLMs [20.084201133669534]
  Multi-agent systems (MAS) and reinforcement learning (RL) are widely used to enhance the agentic capabilities of large language models (LLMs)<n>Applying on-policy RL to MAS remains underexplored and presents unique challenges.<n>We propose AT-GRPO, which includes (i) an agent- and turn-wise grouped RL algorithm tailored to MAS and (ii) a training system that supports both single- and multi-policy regimes.
  arXiv  Detail & Related papers  (2025-10-13T06:55:09Z)
• AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework [76.96794548655292]
  Large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions.<n>Applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms.<n>We present the AgentRL framework for scalable multi-turn, multi-task agentic RL training.
  arXiv  Detail & Related papers  (2025-10-05T13:40:01Z)
• From Supervision to Exploration: What Does Protein Language Model Learn During Reinforcement Learning? [76.288870982181]
  Protein language models (PLMs) have advanced computational protein science through large-scale pretraining and scalable architectures.<n> reinforcement learning (RL) has broadened exploration and enabled precise multi-objective optimization in protein design.<n>We ask if RL improves sampling efficiency and, more importantly, if it reveals capabilities not captured by supervised learning.
  arXiv  Detail & Related papers  (2025-10-02T01:31:10Z)
• SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks [110.20297293596005]
  Large language model (LLM) agents need to perform multi-turn interactions in real-world tasks.<n>Existing multi-turn RL algorithms for optimizing LLM agents fail to perform effective credit assignment over multiple turns while leveraging the generalization capabilities of LLMs.<n>We propose a novel RL algorithm, SWEET-RL, that uses a carefully designed optimization objective to train a critic model with access to additional training-time information.<n>Our experiments demonstrate that SWEET-RL achieves a 6% absolute improvement in success and win rates on ColBench compared to other state-of-the-art multi-turn RL algorithms.
  arXiv  Detail & Related papers  (2025-03-19T17:55:08Z)
• Reinforcing Competitive Multi-Agents for Playing 'So Long Sucker' [0.12234742322758417]
  This paper investigates the strategy game So Long Sucker (SLS) as a novel benchmark for multi-agent reinforcement learning (MARL)<n>We introduce the first publicly available computational framework for SLS, complete with a graphical user interface and benchmarking support for reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2024-11-17T12:38:13Z)
• Knowledge Graph Reasoning with Self-supervised Reinforcement Learning [30.359557545737747]
  We propose a self-supervised pre-training method to warm up the policy network before the RL training stage.<n>In our supervised learning stage, the agent selects actions based on the policy network and learns from generated labels.<n>We show that our SSRL model meets or exceeds current state-of-the-art results on all Hits@k and mean reciprocal rank (MRR) metrics.
  arXiv  Detail & Related papers  (2024-05-22T13:39:33Z)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL [80.10358123795946]
  We develop a framework for building multi-turn RL algorithms for fine-tuning large language models. Our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks.
  arXiv  Detail & Related papers  (2024-02-29T18:45:56Z)
• Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training [43.97565851415018]
  Robust Reinforcement Learning (RL) focuses on improving performances under model errors or adversarial attacks. Most of the existing literature models RARL as a zero-sum simultaneous game with Nash equilibrium as the solution concept. We introduce a novel hierarchical formulation of robust RL - a general-sum Stackelberg game model called RRL-Stack.
  arXiv  Detail & Related papers  (2022-02-19T03:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.