SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees

• URL: http://arxiv.org/abs/2605.05216v1
• Date: Fri, 17 Apr 2026 01:45:30 GMT
• Title: SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees
• Abstract summary: Large language models (LLMs) with a large number of parameters achieve strong performance but are often prohibitively expensive to deploy.<n>Recent work explores using teams of smaller, more efficient LLMs that collectively match or even outperform a single large model.<n>We address this by introducing Sequential Agent Tuning (SAT), a coordinator-free training paradigm.
• Score: 20.52379192411959
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) with a large number of parameters achieve strong performance but are often prohibitively expensive to deploy. Recent work explores using teams of smaller, more efficient LLMs that collectively match or even outperform a single large model. However, jointly updating multiple agents introduces compounding distribution shifts, making coordination and stability during training difficult. We address this by introducing Sequential Agent Tuning (SAT), a coordinator-free training paradigm. SAT represents the team as a factorized policy and employs block-coordinate updates over agents, enabling scalable, decentralized training without a central controller. Specifically, we develop a sequence-aware, on-policy advantage estimator that conditions on the evolving team policy, coupled with per-agent KL trust regions that isolate occupancy drift. Theoretically, this framework provides two critical guarantees. First, it ensures monotonic improvement, stabilizing the training process. Second, it establishes provable plug-and-play invariance: any agent can be upgraded to a stronger model without retraining the rest of the team, with a formal guarantee that the performance bound improves. Empirically, a team of three 4B agents (12B total) trained with SAT surpasses the much larger Qwen3-32B on AIME24/25 benchmarks by 3.9\% on average. We validate our plug-and-play theory by swapping in two 8B agents, which boosts the composite score by 10.4\%. We provide code and appendix of proof at https://github.com/Yydc/SAT-AAMAS

Related papers

• Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning [5.8010446129208155]
  We introduce a clipping objective to control the upper bounds of the advantage fluctuation in sequential updates.<n>We derive two new practical algorithms using our clipping objective.<n>Our proposed method is highlighted with both stable convergence properties and the desired low advantage variance estimation.
  arXiv  Detail & Related papers  (2026-06-24T08:05:11Z)
• CONCAT: Consensus- and Confidence-Driven Ad Hoc Teaming for Efficient LLM-Based Multi-Agent Systems [28.51314301845782]
  CONCAT is a training-free multi-agent collaboration framework based on CONsensus and Confidence-driven Ad hoc Teaming.<n> Experiments across three LLMs and three benchmarks show that CONCAT achieves up to 2.02x higher efficiency (accuracy/latency ratio) than LLM-Debate.
  arXiv  Detail & Related papers  (2026-05-28T08:47:54Z)
• TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination [11.65571332626047]
  TeamTR is a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control.<n>We show that TeamTR outperforms single-agent and sequential baselines with 7.1% on average, mitigates coordination regressions, and supports plug-and-play component replacement.
  arXiv  Detail & Related papers  (2026-05-01T23:42:57Z)
• Small Model as Master Orchestrator: Learning Unified Agent-Tool Orchestration with Parallel Subtask Decomposition [61.291733522717415]
  Agent-as-Tool is a unified parallel orchestration paradigm that mitigates both agents and tools into a standardized, learnable action space.<n>ParaManager decouples planning decisions from subtask solving, enabling state-aware parallel subtask decomposition, delegation, and asynchronous execution.<n>Experiments show that ParaManager achieves strong performance across multiple benchmarks and exhibits robust generalization under unseen model pools.
  arXiv  Detail & Related papers  (2026-04-18T14:41:27Z)
• Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning [68.85408801740228]
  We propose textbfAgent Q-Mix, a reinforcement learning framework that reformulates topology selection as a cooperative Multi-Agent Reinforcement Learning problem.<n>Our method learns decentralized communication decisions using Qmix value factorization, where each agent selects from a set of communication actions that jointly induce a round-wise communication graph.<n>Agent Q-Mix achieves the highest average accuracy compared to existing methods while demonstrating superior token efficiency and robustness against agent failure.
  arXiv  Detail & Related papers  (2026-04-01T00:38:24Z)
• Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning [112.16686518063456]
  We introduce textbfMulti-Agent Test-Time Reinforcement Learning (MATTRL), a framework that injects structured textual experience into multi-agent deliberation at inference time.<n>MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making.<n>Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67% over a multi-agent baseline, and by 8.67% over comparable single-agent baselines
  arXiv  Detail & Related papers  (2026-01-14T17:57:43Z)
• SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent [63.15417992240217]
  We introduce SkyRL-Agent, a framework for efficient, multi-turn, long-horizon agent training and evaluation.<n>It provides efficient asynchronous dispatching, lightweight tool integration, and flexible backend interoperability.<n>We train SA-SWE-32B, a software engineering agent trained from Qwen3-32B (24.4% Pass@1) purely with reinforcement learning.
  arXiv  Detail & Related papers  (2025-11-20T07:05:19Z)
• Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO [24.532870400949424]
  Current training methods train a unified large language model for all agents in the system.<n>This may limit the performances due to different underlying distributions for different agents.<n>We propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization for vertical Multi-agent systems.
  arXiv  Detail & Related papers  (2025-11-17T12:06:30Z)
• Reinforcement Learning for Machine Learning Engineering Agents [52.03168614623642]
  We show that agents backed by weaker models that improve via reinforcement learning can outperform agents backed by much larger, but static models.<n>We propose duration- aware gradient updates in a distributed asynchronous RL framework to amplify high-cost but high-reward actions.<n>We also propose environment instrumentation to offer partial credit, distinguishing almost-correct programs from those that fail early.
  arXiv  Detail & Related papers  (2025-09-01T18:04:10Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Order Matters: Agent-by-agent Policy Optimization [41.017093493743765]
  A sequential scheme that updates policies agent-by-agent provides another perspective and shows strong performance. We propose the textbfAgent-by-textbfagent textbfPolicy textbfOptimization (A2PO) algorithm to improve the sample efficiency.
  arXiv  Detail & Related papers  (2023-02-13T09:24:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.