Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning

• URL: http://arxiv.org/abs/2606.26397v1
• Date: Wed, 24 Jun 2026 21:28:49 GMT
• Title: Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning
• Abstract summary: We introduce a novel preference-conditioned Bellman operator, motivated from the Chebyshev scalarization.<n>We show how to extract deterministic policies from these converged Q-estimates.<n> Experimental results validate that our algorithm successfully recovers complex trade-offs.
• Score: 3.960587756853285
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Real-world decision-making often requires balancing multiple conflicting objectives, a challenge that standard Reinforcement Learning (RL) frequently addresses by aggregating rewards into a single scalar signal. While effective for simple tasks, this approach often fails to capture the full spectrum of optimal trade-offs, known as the Pareto frontier. In this paper, we introduce a novel preference-conditioned Bellman operator, motivated from the Chebyshev scalarization, designed to compute deterministic Pareto-optimal policies for Multi-Objective Markov Decision Processes (MOMDPs). We prove that this operator satisfies an enveloping property, where the estimated value functions upper-bound the true Pareto frontier, and demonstrate that it monotonically converges to a coverage set of this frontier. Furthermore, we also show how to extract deterministic policies from these converged Q-estimates. This ensures the agent can recover a policy for any given preference, capturing the entire Pareto-optimal frontier while guaranteeing each synthesized policy remains approximately Pareto-optimal. Experimental results validate that our algorithm successfully recovers complex trade-offs, providing a solution for deterministic Pareto-optimal policy synthesis.

Related papers

• Exact Model-Free Policy Iteration for Co-safe LTL Planning [2.5735476569508995]
  This work studies model-free reinforcement learning for co-safe linear temporal logic objectives in finite Markov decision processes.<n>We develop a new two-step model-free reinforcement learning method that first uses a discounted surrogate to identify a clamp set.<n>We prove almost-sure convergence of the policy evaluation step and finite termination of the policy iteration algorithm at an optimal policy.
  arXiv  Detail & Related papers  (2026-08-05T16:58:25Z)
• A Noise-Robust Elicit-to-Optimize Framework for Distortion Riskmetrics via Inverse Reinforcement Learning [6.58410442967839]
  We propose a noise-robust elicit-to-optimize framework that integrates inverse reinforcement learning (IRL) and reinforcement learning (RL)<n>On the elicitation side, we propose an adaptive IRL method that infers agents' latent risk objectives from noisy observed decisions.<n>A comprehensive empirical study demonstrates the framework's elicitation accuracy and effectiveness in complex financial environments.
  arXiv  Detail & Related papers  (2026-07-15T21:22:50Z)
• A Single Deep Preference-Conditioned Policy for Learning Pareto Coverage Sets [0.4369550829556578]
  We study the uniqueness and continuity of the preference-to-solution correspondence under nonlinear scalarization.<n>We use smooth Tchebycheff scalarization as a monotone utility to solve the problem.<n>We show that each update is equivalent to solving a Kullback-Leibler-regularized MDP with the previous policy as reference.
  arXiv  Detail & Related papers  (2026-05-09T13:35:50Z)
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards [47.557539197058496]
  We introduce Random Policy Valuation for Diverse Reasoning (ROVER)<n>ROVER is a minimalist yet highly effective RL method that samples actions from a softmax over uniform-policy Q-values.<n>It demonstrates superior performance in both textbfquality (textbf+8.2 on pass@1, textbf+16.8 on pass@256) and textbfdiversity (textbf+17.6%)
  arXiv  Detail & Related papers  (2025-09-29T16:09:07Z)
• Recursive Reward Aggregation [60.51668865089082]
  We propose an alternative approach for flexible behavior alignment that eliminates the need to modify the reward function.<n>By introducing an algebraic perspective on Markov decision processes (MDPs), we show that the Bellman equations naturally emerge from the generation and aggregation of rewards.<n>Our approach applies to both deterministic and deterministic settings and seamlessly integrates with value-based and actor-critic algorithms.
  arXiv  Detail & Related papers  (2025-07-11T12:37:20Z)
• Mirror Descent Policy Optimisation for Robust Constrained Markov Decision Processes [12.666842349236788]
  This paper presents mirror descent policy optimisation for robust constrained Markov decision processes.<n>We make use of policy gradient techniques to optimise both the policy (as a maximiser) and the transition kernel (as an adversarial minimiser) on the Lagrangian.<n>Experiments confirm the benefits of mirror descent policy optimisation in constrained and unconstrained optimisation.
  arXiv  Detail & Related papers  (2025-06-29T09:55:52Z)
• Learning Deterministic Policies with Policy Gradients in Constrained Markov Decision Processes [59.27926064817273]
  We introduce an exploration-agnostic algorithm, called C-PG, which enjoys global last-iterate convergence guarantees under domination assumptions.<n>We empirically validate both the action-based (C-PGAE) and parameter-based (C-PGPE) variants of C-PG on constrained control tasks.
  arXiv  Detail & Related papers  (2025-06-06T10:29:05Z)
• Non-maximizing policies that fulfill multi-criterion aspirations in expectation [0.7874708385247353]
  In dynamic programming and reinforcement learning, the policy for the sequential decision making of an agent is usually determined by expressing the goal as a scalar reward function.<n>We consider finite acyclic Decision Markov Processes with multiple distinct evaluation metrics, which do not necessarily represent quantities that the user wants to be maximized.<n>Our algorithm guarantees that this task is fulfilled by using simplices to approximate feasibility sets and propagate aspirations forward while ensuring they remain feasible.
  arXiv  Detail & Related papers  (2024-08-08T11:41:04Z)
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients [62.81324245896716]
  Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. In common practice, convergence (hyper)policies are learned only to deploy their deterministic version. We show how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy.
  arXiv  Detail & Related papers  (2024-05-03T16:45:15Z)
• Trust-Region-Free Policy Optimization for Stochastic Policies [60.52463923712565]
  We show that the trust region constraint over policies can be safely substituted by a trust-region-free constraint without compromising the underlying monotonic improvement guarantee. We call the resulting algorithm Trust-REgion-Free Policy Optimization (TREFree) explicit as it is free of any trust region constraints.
  arXiv  Detail & Related papers  (2023-02-15T23:10:06Z)
• Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy [95.98698822755227]
  We make the first attempt to study risk-sensitive deep reinforcement learning under the average reward setting with the variance risk criteria. We propose an actor-critic algorithm that iteratively and efficiently updates the policy, the Lagrange multiplier, and the Fenchel dual variable.
  arXiv  Detail & Related papers  (2020-12-28T05:02:26Z)
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking [59.691919635037216]
  Policy evaluation is a key process in Reinforcement Learning (RL) We devise an optimization method, called Kalman Optimization for Value Approximation (KOVA) KOVA minimizes a regularized objective function that concerns both parameter and noisy return uncertainties.
  arXiv  Detail & Related papers  (2020-02-17T13:30:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.