Multi-Agent Reinforcement Learning via Agent-Specific Preference

• URL: http://arxiv.org/abs/2608.08604v1
• Date: Sun, 09 Aug 2026 09:38:41 GMT
• Title: Multi-Agent Reinforcement Learning via Agent-Specific Preference
• Abstract summary: Multi-agent reinforcement learning relies heavily on well-defined global reward functions.<n>We introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling.<n>We show that MAGPIE achieves performance comparable to reward-engineered baselines.
• Score: 13.390117775984798
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions. Designing such rewards is challenging, especially in systems with heterogeneous agents, where a single scalar objective may fail to capture diverse behaviors. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling. Each agent is evaluated by a dedicated expert through preference signals, eliminating the need for global evaluation. We theoretically prove that optimizing these decentralized preferences converges to a Nash equilibrium policy. To integrate local preferences into a coherent global objective, we construct agent-specific reward models from preference data and combine them via a monotonic aggregation mechanism. We further prove that optimizing this aggregate reward model is equivalent to training the Nash equilibrium policy. Extensive experiments on benchmark multi-agent tasks and a sequential production line task show that MAGPIE achieves performance comparable to reward-engineered baselines, demonstrating its potential to facilitate policy learning in scenarios where precise reward engineering is impractical.

Related papers

• Poly-EPO: Training Exploratory Reasoning Models [62.82992914206963]
  We present a framework for post-training language models (LMs) that explicitly encourages optimistic exploration and promotes a synergy between exploration and exploitation.<n>We propose Polychromic Exploratory Policy Optimization (Poly-EPO), which instantiates this framework with an objective that explicitly synergizes exploration and exploitation.
  arXiv  Detail & Related papers  (2026-04-19T22:54:19Z)
• CoFi-PGMA: Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs [0.0]
  We introduce CoFi-PGMA, a unified framework for learning under filtered feedback in multi-agent LLM systems.<n>Our approach derives a counterfactual per-agent training objective based on marginal contribution, which corrects the learning signal under both routing and collaborative mechanisms.
  arXiv  Detail & Related papers  (2026-04-03T22:55:23Z)
• Counterfactual Credit Policy Optimization for Multi-Agent Collaboration [27.260904103460664]
  Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles and aggregating diverse hypotheses.<n>We introduce Counterfactual Credit Policy Optimization (CCPO), a framework that assigns agent-specific learning signals.<n> CCPO builds dynamic counterfactual baselines that simulate outcomes with an agent's contribution removed.
  arXiv  Detail & Related papers  (2026-03-23T04:35:02Z)
• ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning [88.87305027895657]
  This work investigates Reinforcement Learning-based optimization for shopping agents in real-world scenarios.<n>Agents must simultaneously satisfy multiple interdependent objectives spanning objective metrics (product correctness), subjective qualities (persuasiveness), outcome rewards (final response quality), and process rewards (tool efficiency)<n>Experiments demonstrate that our RL-trained agent, ChatShopBuddy, consistently outperforms larger models relying on generic reasoning.
  arXiv  Detail & Related papers  (2026-03-06T09:18:51Z)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning [50.93295951454092]
  We introduce a set level diversity objective defined over sampled trajectories using kernelized similarity.<n>Our approach derives a leave-one-out marginal contribution for each sampled trajectory and integrates this objective as a plug-in advantage shaping term for policy optimization.<n>Experiments across a range of model scales demonstrate the effectiveness of our proposed algorithm, consistently outperforming strong baselines in both Pass@1 and Pass@K across various benchmarks.
  arXiv  Detail & Related papers  (2026-02-01T07:13:20Z)
• Multimodal Reinforcement Learning with Agentic Verifier for AI Agents [131.46008226323423]
  Argos is a principled multimodal reward agent to train reasoning models for agentic tasks.<n>By leveraging our agentic verifier across both SFT data and RL training, our model achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2025-12-03T04:42:47Z)
• Achieving Equilibrium under Utility Heterogeneity: An Agent-Attention Framework for Multi-Agent Multi-Objective Reinforcement Learning [30.138706163658597]
  We propose an Agent-Attention Multi-Agent Multi-Objective Reinforcement Learning (AA-MAMORL) framework.<n>Our approach implicitly learns a joint belief over other agents' utility functions and their associated policies during training.<n>In execution, each agent independently selects actions based on local observations and its private utility function to approximate a BNE.
  arXiv  Detail & Related papers  (2025-11-12T03:06:21Z)
• Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents [28.145430029174577]
  Large language model (LLM)-based agents are increasingly trained with reinforcement learning (RL) to enhance their ability to interact with external environments.<n>Existing approaches typically rely on outcome-based rewards that are only provided at the final answer.<n>In this paper, we propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training.
  arXiv  Detail & Related papers  (2025-10-16T17:59:32Z)
• Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment [1.8552770604791606]
  We propose a hybrid reward modeling framework that integrates complementary reward paradigms.<n>We show consistent improvements across different multimodal benchmarks when applying hybrid and multi-aspect reward modeling.<n>Our best performing model in the 3B family achieves an overall average improvement of 9.5% across general and math reasoning tasks.
  arXiv  Detail & Related papers  (2025-10-06T18:53:23Z)
• Recursive Reward Aggregation [60.51668865089082]
  We propose an alternative approach for flexible behavior alignment that eliminates the need to modify the reward function.<n>By introducing an algebraic perspective on Markov decision processes (MDPs), we show that the Bellman equations naturally emerge from the generation and aggregation of rewards.<n>Our approach applies to both deterministic and deterministic settings and seamlessly integrates with value-based and actor-critic algorithms.
  arXiv  Detail & Related papers  (2025-07-11T12:37:20Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.