Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning

• URL: http://arxiv.org/abs/2606.18111v1
• Date: Tue, 16 Jun 2026 16:16:54 GMT
• Title: Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning
• Abstract summary: We formalize the fair optimization problem in multi-policy MORL.<n>We show that for concave, piecewise-linear welfare functions, fair policies remain in the convex coverage set.<n>We demonstrate that non-stationary policies, augmented with accrued reward histories, and state policies improve fairness by dynamically adapting to historical inequities.
• Score: 1.3483705601540006
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Fairness is an important aspect of decision-making in multi-objective reinforcement learning (MORL), where policies must ensure both optimality and equity across multiple, potentially conflicting objectives. While single-policy MORL methods can learn fair policies for fixed user preferences using welfare functions such as the generalized Gini welfare function (GGF), they fail to provide the diverse set of policies necessary for dynamic or unknown user preferences. To address this limitation, we formalize the fair optimization problem in multi-policy MORL, where the goal is to learn a set of Pareto-optimal policies that ensure fairness across all possible user preferences. Our key technical contributions are threefold: (1) We show that for concave, piecewise-linear welfare functions (e.g., GGF), fair policies remain in the convex coverage set (CCS), which is an approximated Pareto front for linear scalarization. (2) We demonstrate that non-stationary policies, augmented with accrued reward histories, and stochastic policies improve fairness by dynamically adapting to historical inequities. (3) We propose three novel algorithms, which include integrating GGF with multi-policy multi-objective Q-Learning (MOQL), state-augmented multi-policy MOQL for learning non-statoinary policies, and its novel extension for learning stochastic policies. We evaluate our algorithms across various domains and compare our methods against the state-of-the-art MORL baselines. The empirical results show that our methods learn a set of fair policies that accommodate different user preferences.

Related papers

• PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models [56.37206340175191]
  Large Language Models (LLMs) are increasingly integrated into real-world decision-making.<n>Yet, their ability to comprehend and reason about policy-related content remains underexplored.<n>We present textbftextitPolicyBench, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension.
  arXiv  Detail & Related papers  (2026-04-14T17:27:50Z)
• Polychromic Objectives for Reinforcement Learning [63.37185057794815]
  Reinforcement learning fine-tuning (RLFT) is a dominant paradigm for improving pretrained policies for downstream tasks.<n>We introduce an objective for policy methods that explicitly enforces the exploration and refinement of diverse generations.<n>We show how proximal policy optimization (PPO) can be adapted to optimize this objective.
  arXiv  Detail & Related papers  (2025-09-29T19:32:11Z)
• Residual Policy Gradient: A Reward View of KL-regularized Objective [48.39829592175419]
  Reinforcement Learning and Imitation Learning have achieved widespread success in many domains but remain constrained during real-world deployment.<n>Policy customization has been introduced, aiming to adapt a prior policy while preserving its inherent properties and meeting new task-specific requirements.<n>A principled approach to policy customization is Residual Q-Learning (RQL), which formulates the problem as a Markov Decision Process (MDP) and derives a family of value-based learning algorithms.<n>We introduce Residual Policy Gradient (RPG), which extends RQL to policy gradient methods, allowing policy customization in gradient-based RL settings.
  arXiv  Detail & Related papers  (2025-03-14T02:30:13Z)
• Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation [6.876580618014666]
  In this paper, we adapt inverse reinforcement learning (IRL) by using reward distance estimates for regularizing the discriminator. We show that ParIRL outperforms other IRL algorithms for various multi-objective control tasks.
  arXiv  Detail & Related papers  (2024-08-22T03:51:39Z)
• Personalized Reinforcement Learning with a Budget of Policies [9.846353643883443]
  Personalization in machine learning (ML) tailors models' decisions to the individual characteristics of users. We propose a novel framework termed represented Markov Decision Processes (r-MDPs) that is designed to balance the need for personalization with the regulatory constraints. In an r-MDP, we cater to a diverse user population, each with unique preferences, through interaction with a small set of representative policies. We develop two deep reinforcement learning algorithms that efficiently solve r-MDPs.
  arXiv  Detail & Related papers  (2024-01-12T11:27:55Z)
• Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning [13.627087954965695]
  We propose a human-in-the-loop policy optimization framework for preference-based MORL. Our method proactively learns the DM's implicit preference information without requiring any priori knowledge. We evaluate our approach against three conventional MORL algorithms and four state-of-the-art preference-based MORL algorithms.
  arXiv  Detail & Related papers  (2024-01-04T09:17:53Z)
• Fairness in Preference-based Reinforcement Learning [2.3388338598125196]
  We design a new fairness-induced preference-based reinforcement learning or FPbRL. The main idea of FPbRL is to learn vector reward functions associated with multiple objectives via new welfare-based preferences. Experiment studies show that the proposed FPbRL approach can achieve both efficiency and equity for learning effective and fair policies.
  arXiv  Detail & Related papers  (2023-06-16T17:47:36Z)
• Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer [7.970144204429356]
  We introduce an SF-based extension of the Optimistic Linear Support algorithm to learn a set of policies whose SFs form a convex coverage set. We prove that policies in this set can be combined via generalized policy improvement to construct optimal behaviors for any new linearly-expressible tasks.
  arXiv  Detail & Related papers  (2022-06-22T19:00:08Z)
• Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates [63.58053355357644]
  We study the problem of learning a good set of policies, so that when combined together, they can solve a wide variety of unseen reinforcement learning tasks. We show theoretically that having access to a specific set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance.
  arXiv  Detail & Related papers  (2021-12-30T12:20:46Z)
• Imitation Learning from MPC for Quadrupedal Multi-Gait Control [63.617157490920505]
  We present a learning algorithm for training a single policy that imitates multiple gaits of a walking robot. We use and extend MPC-Net, which is an Imitation Learning approach guided by Model Predictive Control. We validate our approach on hardware and show that a single learned policy can replace its teacher to control multiple gaits.
  arXiv  Detail & Related papers  (2021-03-26T08:48:53Z)
• Variational Policy Propagation for Multi-agent Reinforcement Learning [68.26579560607597]
  We propose a emphcollaborative multi-agent reinforcement learning algorithm named variational policy propagation (VPP) to learn a emphjoint policy through the interactions over agents. We prove that the joint policy is a Markov Random Field under some mild conditions, which in turn reduces the policy space effectively. We integrate the variational inference as special differentiable layers in policy such as the actions can be efficiently sampled from the Markov Random Field and the overall policy is differentiable.
  arXiv  Detail & Related papers  (2020-04-19T15:42:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.