Entropy-Augmented Multi-Objective Policy Optimization in Multiagent Systems

• URL: http://arxiv.org/abs/2608.12534v1
• Date: Wed, 12 Aug 2026 19:07:47 GMT
• Title: Entropy-Augmented Multi-Objective Policy Optimization in Multiagent Systems
• Abstract summary: Multi-objective evolutionary algorithms such as NSGA-II optimize for diversity in the objective space, but neglect diversity in the behavior space.<n>We introduce an entropy-augmented policy evaluation strategy that incorporates an entropy bonus into agent fitness scores.<n>We observe hypervolume improvements of up to 48% relative to the NSGA-II baseline.
• Score: 0.9799637101641152
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous agent teams deployed in settings such as marine and extraterrestrial outposts must coordinate actions to achieve optimal outcomes across multiple competing objectives. Multi-objective evolutionary algorithms such as NSGA-II optimize for diversity in the objective space, but neglect diversity in the behavior space, possibly leading to premature convergence and a collapse in behaviors that may differentiate policies in different external conditions. To address this, we introduce an entropy-augmented policy evaluation strategy that incorporates an entropy bonus into agent fitness scores, discouraging behavioral homogeneity across the evolving population. By augmenting policy evaluation with a behavior-space diversity signal while preserving the underlying Pareto optimization framework, our method is designed to encourage exploration of behaviorally distinct policies in multiagent domains. We evaluate our approach across rover-domain experiments with qualitatively distinct reward structures and observe hypervolume improvements of up to 48% relative to the NSGA-II baseline, suggesting that behavioral diversity is a promising and underexplored direction for improving multi-objective multiagent evolutionary optimization.

Related papers

• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning [50.93295951454092]
  We introduce a set level diversity objective defined over sampled trajectories using kernelized similarity.<n>Our approach derives a leave-one-out marginal contribution for each sampled trajectory and integrates this objective as a plug-in advantage shaping term for policy optimization.<n>Experiments across a range of model scales demonstrate the effectiveness of our proposed algorithm, consistently outperforming strong baselines in both Pass@1 and Pass@K across various benchmarks.
  arXiv  Detail & Related papers  (2026-02-01T07:13:20Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• Evolutionary Policy Optimization [47.30139909878251]
  On-policy reinforcement learning (RL) algorithms are widely used for their strong performance and training stability, but they struggle to scale with larger batch sizes.<n>We propose Evolutionary Policy Optimization (EPO), a hybrid that combines the scalability and diversity of EAs with the performance and stability of policy gradients.
  arXiv  Detail & Related papers  (2025-03-24T18:08:54Z)
• Preference-Guided Diffusion for Multi-Objective Offline Optimization [64.08326521234228]
  We propose a preference-guided diffusion model for offline multi-objective optimization.<n>Our guidance is a preference model trained to predict the probability that one design dominates another.<n>Our results highlight the effectiveness of classifier-guided diffusion models in generating diverse and high-quality solutions.
  arXiv  Detail & Related papers  (2025-03-21T16:49:38Z)
• Phasic Diversity Optimization for Population-Based Reinforcement Learning [10.15130620537703]
  Phasic Diversity Optimization (PDO) algorithm separates reward and diversity training into distinct phases. In the auxiliary phase, agents with poor performance diversified via determinants will not replace the better agents in the archive. We introduce two implementations of PDO archive and conduct tests in the newly proposed adversarial dogfight and MuJoCo simulations.
  arXiv  Detail & Related papers  (2024-03-17T06:41:09Z)
• Knowledge Transfer for Dynamic Multi-objective Optimization with a Changing Number of Objectives [4.490459770205671]
  We show that the state-of-the-art transfer algorithm for DMOPs with a changing number of objectives lacks sufficient diversity. We propose a knowledge transfer dynamic multi-objective evolutionary algorithm (KTDMOEA) to enhance population diversity after changes.
  arXiv  Detail & Related papers  (2023-06-19T01:54:44Z)
• Multi-Value Alignment in Normative Multi-Agent System: Evolutionary Optimisation Approach [1.160208922584163]
  This research proposes a multi-value promotion model that uses multi-objective evolutionary algorithms to produce the optimum parametric set of norms. Several evolutionary algorithms were used to find a set of optimised norm parameters considering two toy tax scenarios with two and five values are considered.
  arXiv  Detail & Related papers  (2023-05-12T10:30:20Z)
• Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning [139.53668999720605]
  We present a multi-agent PPO algorithm in which the local policy of each agent is updated similarly to vanilla PPO. We prove that with standard regularity conditions on the Markov game and problem-dependent quantities, our algorithm converges to the globally optimal policy at a sublinear rate.
  arXiv  Detail & Related papers  (2023-05-08T16:20:03Z)
• Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach [128.62787284435007]
  We propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation-invariant actor-critic neural architecture. We prove that MF-PPO attains the globally optimal policy at a sublinear rate of convergence. In particular, we show that the inductive bias introduced by the permutation-invariant neural architecture enables MF-PPO to outperform existing competitors.
  arXiv  Detail & Related papers  (2021-05-18T04:35:41Z)
• Niching Diversity Estimation for Multi-modal Multi-objective Optimization [9.584279193016522]
  Niching is an important and widely used technique in evolutionary multi-objective optimization. In MMOPs, a solution in the objective space may have multiple inverse images in the decision space, which are termed as equivalent solutions. A general niching mechanism is proposed to make standard diversity estimators more efficient when handling MMOPs.
  arXiv  Detail & Related papers  (2021-01-31T05:23:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.