SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs

• URL: http://arxiv.org/abs/2609.08452v1
• Date: Tue, 08 Sep 2026 08:52:23 GMT
• Title: SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs
• Abstract summary: Multi-agent large language models solve complex tasks by coordinating several policies in a shared environment.<n>Existing reinforcement learning methods usually optimize each response or trajectory separately, even when several outputs jointly cause one state transition.<n>We propose SRPO, which treats the active set the minimal set of outputs consumed by one transition, as one multi-agent action.
• Score: 16.941911451861277
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent large language models solve complex tasks by coordinating several policies in a shared environment. However, existing reinforcement learning methods usually optimize each response or trajectory separately, even when several outputs jointly cause one state transition. Consequently, the update unit differs from the action executed by the system. To address this problem, we propose SRPO (Setwise Relative Policy Optimization), which treats the active set the minimal set of outputs consumed by one transition, as one multi-agent action. Specifically, SRPO combines member log-ratios into one cardinality-normalized set ratio, assigns one relative advantage, and clips the set once. This formulation unifies division of labor and joint co-evolution as actions with different set sizes. Experiments on mathematical reasoning and multi-turn search demonstrate one training interface for fixed, mixed, and dynamically routed workflows across four model scales, with the strongest macro-average results among the reported comparisons. Optimization diagnostics further characterize its stability under different event reductions and set sizes.

Related papers

• Contextual Plackett-Luce: An Efficient Neural Model for Probabilistic Sequence Selection under Ambiguity [4.162695034027842]
  Contextual Plackett-Luce is a structured probabilistic model for sequence selection.<n>It can be viewed as a hybrid between fully autoregressive prediction and parallel sequence selection.<n>It achieves improved structural consistency and under ambiguous supervision compared to strong parallel baselines.
  arXiv  Detail & Related papers  (2026-05-09T18:52:53Z)
• Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs [62.17306142810532]
  We present Chimera, a predictive scheduling system for multi-agent workflow serving on heterogeneous LLM clusters.<n>Chime traces the best latency, reducing end-to-end latency by 1.2--2.4$times$ and improving task performance by 8.0-9.5 percentage points on average over competitive baselines.
  arXiv  Detail & Related papers  (2026-03-23T17:01:42Z)
• Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models [67.45032003041399]
  We propose a novel Multi-Paradigm Collaborative Attack (MPCAttack) framework to boost the transferability of adversarial examples against MLLMs.<n>MPCO adaptively balances the importance of different paradigm representations and guides the global optimisation.<n>Our solution consistently outperforms state-of-the-art methods in both targeted and untargeted attacks on open-source and closed-source MLLMs.
  arXiv  Detail & Related papers  (2026-03-05T06:01:26Z)
• Workflow-R1: Group Sub-sequence Policy Optimization for Multi-turn Workflow Construction [25.928675237308074]
  We present gradient-R1, a framework that reformulates workflow construction as a multi-turn, natural language-based sequential decision-making process.<n>GSsPO functions as a structure-aware RL algorithm generalizable to a broad class of multi-turn agentic sequential decision-making tasks.
  arXiv  Detail & Related papers  (2026-02-01T12:44:59Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents [56.625878022978945]
  Large Language Models (LLMs) as autonomous agents are increasingly tasked with solving complex, long-horizon problems.<n>Direct Preference Optimization (DPO) provides a signal that is too coarse for precise credit assignment, while step-level DPO is often too myopic to capture the value of multi-step behaviors.<n>We introduce Hierarchical Preference Learning (HPL), a hierarchical framework that optimize LLM agents by leveraging preference signals at multiple, synergistic granularities.
  arXiv  Detail & Related papers  (2025-09-26T08:43:39Z)
• VFP: Variational Flow-Matching Policy for Multi-Modal Robot Manipulation [3.986404588605909]
  Variational Flow-Matching Policy (VFP) is a flow-matching policy that captures both task-level and trajectory-level multi-modality.<n>VFP achieves a 49% relative improvement in task success rate over standard flow-based baselines in simulation.
  arXiv  Detail & Related papers  (2025-08-03T07:23:02Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)
• Low-Rank Agent-Specific Adaptation (LoRASA) for Multi-Agent Policy Learning [3.333453555166201]
  Multi-agent reinforcement learning (MARL) often relies on emph parameter sharing (PS) to scale efficiently.<n>We propose textbfLow-Rank Agent-Specific Adaptation (LoRASA), a novel approach that treats each agent's policy as a specialized task'' fine-tuned from a shared backbone.<n>We evaluate LoRASA on challenging benchmarks including the StarCraft Multi-Agent Challenge (SMAC) and Multi-Agent MuJoCo (MAMuJoCo)
  arXiv  Detail & Related papers  (2025-02-08T13:57:53Z)
• CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition [10.045163723630159]
  CHASE operates as a sample-adaptive normalization method to mitigate inter-entity distribution discrepancies.<n>Our approach seamlessly adapts to single-entity backbones and boosts their performance in multi-entity scenarios.
  arXiv  Detail & Related papers  (2024-10-09T17:55:43Z)
• UCB-driven Utility Function Search for Multi-objective Reinforcement Learning [51.00436121587591]
  In Multi-objective Reinforcement Learning (MORL) agents are tasked with optimising decision-making behaviours.<n>We focus on the case of linear utility functions parametrised by weight vectors w.<n>We introduce a method based on Upper Confidence Bound to efficiently search for the most promising weight vectors during different stages of the learning process.
  arXiv  Detail & Related papers  (2024-05-01T09:34:42Z)
• Asynchronous, Option-Based Multi-Agent Policy Gradient: A Conditional Reasoning Approach [10.904610735933145]
  Multi-agent policy gradient (MAPG) methods are commonly used to learn such policies. In complex problems with large state and action spaces, it is advantageous to extend MAPG methods to use higher-level actions. We propose a novel, conditional reasoning approach to address this problem.
  arXiv  Detail & Related papers  (2022-03-29T22:02:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.