Training Small LLMs as Spatial Multi-Agent Policies

• URL: http://arxiv.org/abs/2608.01425v1
• Date: Sun, 02 Aug 2026 18:14:40 GMT
• Title: Training Small LLMs as Spatial Multi-Agent Policies
• Abstract summary: We take up both threads in spatial cooperative games, where small frozen LLMs prompt with low-level actions fail outright, earning zero reward.<n>We equip each game with a library of symbolic emphoptions: typed, state-feasible, short-horizon behaviors executed by a symbolic planner.<n>Each agent's LLM acts as its policy over options, with a private per-agent LoRA adapter trained by a per-agent variant of multi-agent GRPO.
• Score: 17.254137262162505
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training LLM-based multi-agent systems with multi-agent reinforcement learning is rapidly gaining traction, and a parallel line of work argues that such systems should be judged by their behavior, not only their reward. We take up both threads in spatial cooperative games, where small frozen LLMs prompted with low-level actions fail outright, earning zero reward. Guided by the options/semi-MDP framework---and, because option execution is asynchronous across agents, its multi-agent extension in macro-action Dec-POMDPs---we equip each game with a library of symbolic \emph{options}: typed, state-feasible, short-horizon behaviors executed by a symbolic planner. Each library is drafted by a frontier coding model from the game's source code; the feasibility guards that filter each menu are then synthesized mechanically from cheap random-policy burn-in rollouts---a guard is adopted only if it explains repeated execution failures while hiding no logged success---so no guard is authored, selected, or reward-tuned by hand. Each agent's LLM acts as its policy over options, with a private per-agent LoRA adapter trained by a per-agent variant of multi-agent GRPO (PA-MAGRPO); this lifts frozen bases from zero reward to competent play across three games and four small backbones. Behavioral audits then reveal that reward and cooperation decouple: a rising reward curve may simply mean that one agent has learned to run the entire task alone while its partner idles---cooperation emerges only when the task makes it necessary. Reward alone is thus an unreliable readout of cooperation; behavioral evaluation must sit alongside it.

Related papers

• Is Per-Agent Policy Composition Safe? Rethinking Successor-Feature Transfer in Cooperative Multi-Agent Reinforcement Learning [5.703301837612397]
  We show that multi-agent transfer can produce joint behavior strictly worse than every policy in the library.<n>We propose MA-USFA, a hierarchical method with two layers: a lower layer of universal successor feature approximators that predicts each agent's successor features while conditioned on its teammates' objectives, and an upper composer that selects, across agents, which library entry each agent should follow and supplies the cross-agent correction a per-agent value cannot represent.
  arXiv  Detail & Related papers  (2026-08-12T04:56:52Z)
• CoAgent: Concurrency Control for Multi-Agent Systems [3.127039011380953]
  This paper builds control on a capability classical transactions lack.<n>It fixes a serialization order at launch, serves each read the order-filtered value, and applies writes speculatively in place.<n>It grows a 25-tool library online and lifts the task pass rate from 45/71 to 63/71 at $0.80times$ the time and $0.86times$ the cost.
  arXiv  Detail & Related papers  (2026-06-13T16:15:14Z)
• Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [48.1363632826625]
  Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world.<n>We introduce $alem$, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics.
  arXiv  Detail & Related papers  (2026-06-06T21:13:43Z)
• CAPO: Counterfactual Credit Assignment in Sequential Cooperative Teams [6.8920524670882815]
  We introduce the Sequential Aristocrat Utility (SeqAU), the unique per-agent learning signal.<n>We derive CAPO, a critic-free policy-gradient algorithm.
  arXiv  Detail & Related papers  (2026-04-20T01:14:59Z)
• CoFi-PGMA: Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs [0.0]
  We introduce CoFi-PGMA, a unified framework for learning under filtered feedback in multi-agent LLM systems.<n>Our approach derives a counterfactual per-agent training objective based on marginal contribution, which corrects the learning signal under both routing and collaborative mechanisms.
  arXiv  Detail & Related papers  (2026-04-03T22:55:23Z)
• Multi-Agent Tool-Integrated Policy Optimization [67.12841355267678]
  Large language models (LLMs) increasingly rely on multi-turn tool-integrated planning for knowledge-intensive and complex reasoning tasks.<n>Existing implementations typically rely on a single agent, but they suffer from limited context length and noisy tool responses.<n>No existing methods support effective reinforcement learning post-training of tool-integrated multi-agent frameworks.
  arXiv  Detail & Related papers  (2025-10-06T10:44:04Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)
• Multi-Agent Inverse Q-Learning from Demonstrations [3.4136908117644698]
  Multi-Agent Marginal Q-Learning from Demonstrations (MAMQL) is a novel sample-efficient framework for multi-agent IRL.<n>We show MAMQL significantly outperforms previous multi-agent methods in average reward, sample efficiency, and reward recovery by often more than 2-5x.
  arXiv  Detail & Related papers  (2025-03-06T18:22:29Z)
• MALT: Improving Reasoning with Multi-Agent LLM Training [67.76186488361685]
  MALT (Multi-Agent LLM Training) is a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps.<n>On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively.
  arXiv  Detail & Related papers  (2024-12-02T19:30:36Z)
• Learning From Good Trajectories in Offline Multi-Agent Reinforcement Learning [98.07495732562654]
  offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets. One agent learned by offline MARL often inherits this random policy, jeopardizing the performance of the entire team. We propose a novel framework called Shared Individual Trajectories (SIT) to address this problem.
  arXiv  Detail & Related papers  (2022-11-28T18:11:26Z)
• Off-Beat Multi-Agent Reinforcement Learning [62.833358249873704]
  We investigate model-free multi-agent reinforcement learning (MARL) in environments where off-beat actions are prevalent. We propose a novel episodic memory, LeGEM, for model-free MARL algorithms. We evaluate LeGEM on various multi-agent scenarios with off-beat actions, including Stag-Hunter Game, Quarry Game, Afforestation Game, and StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2022-05-27T02:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.