The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm

• URL: http://arxiv.org/abs/2406.07826v1
• Date: Wed, 12 Jun 2024 02:47:54 GMT
• Title: The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm
• Authors: Giseung Park, Woohyeon Byeon, Seongmin Kim, Elad Havakuk, Amir Leshem, Youngchul Sung,
• Abstract summary: We consider multi-objective reinforcement learning, which arises in many real-world problems with multiple optimization goals. We develop a relevant theory and a practical model-free algorithm under the max-min framework.
• Score: 21.36281978932632
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we consider multi-objective reinforcement learning, which arises in many real-world problems with multiple optimization goals. We approach the problem with a max-min framework focusing on fairness among the multiple goals and develop a relevant theory and a practical model-free algorithm under the max-min framework. The developed theory provides a theoretical advance in multi-objective reinforcement learning, and the proposed algorithm demonstrates a notable performance improvement over existing baseline methods.

Related papers

• Analyzing the Impact of Multimodal Perception on Sample Complexity and Optimization Landscapes in Imitation Learning [0.0]
  We show that properly integrated multimodal policies can achieve tighter generalization bounds and more favorable optimization landscapes than their unimodal counterparts.<n>We provide a comprehensive review of theoretical frameworks that explain why multimodal architectures like PerAct and CLIPort achieve superior performance.
  arXiv  Detail & Related papers  (2025-08-07T07:01:53Z)
• VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning [69.44871115752055]
  We propose an advanced multimodal reasoning model trained via a novel Progressive Curriculum Reinforcement Learning (PCuRL) framework.<n>PCuRL systematically guides the model through tasks of gradually increasing difficulty, substantially improving its reasoning abilities across diverse multimodal contexts.<n>The framework introduces two key innovations: (1) an online difficulty soft weighting mechanism, dynamically adjusting training difficulty across successive RL training stages; and (2) a dynamic length reward mechanism, which encourages the model to adaptively regulate its reasoning path length according to task complexity.
  arXiv  Detail & Related papers  (2025-07-30T12:23:21Z)
• Large Language Models as Computable Approximations to Solomonoff Induction [11.811838796672369]
  We establish the first formal connection between large language models (LLMs) and Algorithmic Information Theory (AIT)<n>We leverage AIT to provide a unified theoretical explanation for in-context learning, few-shot learning, and scaling laws.<n>Our framework bridges the gap between theoretical foundations and practical LLM behaviors, providing both explanatory power and actionable insights for future model development.
  arXiv  Detail & Related papers  (2025-05-21T17:35:08Z)
• Aligned Multi Objective Optimization [15.404668020811513]
  In machine learning practice, there are many scenarios where such conflict does not take place. Recent findings from multi-task learning, reinforcement learning, and LLMs training show that diverse related tasks can enhance performance across objectives simultaneously. We introduce the Aligned Multi-Objective Optimization framework, propose new algorithms for this setting, and provide theoretical guarantees of their superior performance.
  arXiv  Detail & Related papers  (2025-02-19T20:50:03Z)
• Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond [35.78910104369677]
  This paper provides a comprehensive survey of gradient-based techniques for multi-objective deep learning.<n>We systematically categorize existing algorithms based on their outputs.<n>In addition to this taxonomy, the survey covers theoretical analyses, key applications, practical resources, and highlights open challenges and promising directions for future research.
  arXiv  Detail & Related papers  (2025-01-19T04:56:55Z)
• Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought [61.588465852846646]
  Chain-of-Thought (CoT) reasoning has emerged as a promising approach for enhancing the performance of large language models (LLMs) In this work, we introduce a novel reasoning boundary framework (RBF) to address these challenges.
  arXiv  Detail & Related papers  (2024-10-08T05:26:28Z)
• Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques [65.55451717632317]
  We study Multi-Agent Reinforcement Learning from Human Feedback (MARLHF), exploring both theoretical foundations and empirical validations. We define the task as identifying Nash equilibrium from a preference-only offline dataset in general-sum games. Our findings underscore the multifaceted approach required for MARLHF, paving the way for effective preference-based multi-agent systems.
  arXiv  Detail & Related papers  (2024-09-01T13:14:41Z)
• Coding for Intelligence from the Perspective of Category [66.14012258680992]
  Coding targets compressing and reconstructing data, and intelligence. Recent trends demonstrate the potential homogeneity of these two fields. We propose a novel problem of Coding for Intelligence from the category theory view.
  arXiv  Detail & Related papers  (2024-07-01T07:05:44Z)
• A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms [7.081523472610874]
  We devise a control-theoretic reinforcement learning approach to support direct learning of the optimal policy. We empirically evaluate our approach on several classical reinforcement learning tasks.
  arXiv  Detail & Related papers  (2024-06-20T21:50:46Z)
• Towards Principled Task Grouping for Multi-Task Learning [14.3385939018772]
  We present a novel approach to task grouping in Multitask Learning (MTL) Our approach offers a more theoretically grounded method that does not rely on restrictive assumptions for constructing transfer gains.
  arXiv  Detail & Related papers  (2024-02-23T13:51:20Z)
• Hierarchical Optimization-Derived Learning [58.69200830655009]
  We establish a new framework, named Hierarchical ODL (HODL), to simultaneously investigate the intrinsic behaviors of optimization-derived model construction and its corresponding learning process. This is the first theoretical guarantee for these two coupled ODL components: optimization and learning.
  arXiv  Detail & Related papers  (2023-02-11T03:35:13Z)
• Multi-Objective Policy Gradients with Topological Constraints [108.10241442630289]
  We present a new algorithm for a policy gradient in TMDPs by a simple extension of the proximal policy optimization (PPO) algorithm. We demonstrate this on a real-world multiple-objective navigation problem with an arbitrary ordering of objectives both in simulation and on a real robot.
  arXiv  Detail & Related papers  (2022-09-15T07:22:58Z)
• MODRL/D-EL: Multiobjective Deep Reinforcement Learning with Evolutionary Learning for Multiobjective Optimization [10.614594804236893]
  This paper proposes a multiobjective deep reinforcement learning with evolutionary learning algorithm for a typical complex problem called the multiobjective vehicle routing problem with time windows. The experimental results on MO-VRPTW instances demonstrate the superiority of the proposed algorithm over other learning-based and iterative-based approaches.
  arXiv  Detail & Related papers  (2021-07-16T15:22:20Z)
• Discriminator Augmented Model-Based Reinforcement Learning [47.094522301093775]
  It is common in practice for the learned model to be inaccurate, impairing planning and leading to poor performance. This paper aims to improve planning with an importance sampling framework that accounts for discrepancy between the true and learned dynamics.
  arXiv  Detail & Related papers  (2021-03-24T06:01:55Z)
• Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond [114.39616146985001]
  In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of Bi-Level Optimization (BLO) Then we construct a value-function-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies.
  arXiv  Detail & Related papers  (2021-01-27T16:20:23Z)
• Provable Multi-Objective Reinforcement Learning with Generative Models [98.19879408649848]
  We study the problem of single policy MORL, which learns an optimal policy given the preference of objectives. Existing methods require strong assumptions such as exact knowledge of the multi-objective decision process. We propose a new algorithm called model-based envelop value (EVI) which generalizes the enveloped multi-objective $Q$-learning algorithm.
  arXiv  Detail & Related papers  (2020-11-19T22:35:31Z)
• Improving Few-Shot Learning through Multi-task Representation Learning Theory [14.8429503385929]
  We consider the framework of multi-task representation (MTR) learning where the goal is to use source tasks to learn a representation that reduces the sample complexity of solving a target task. We show that recent advances in MTR theory can provide novel insights for popular meta-learning algorithms when analyzed within this framework. This is the first contribution that puts the most recent learning bounds of MTR theory into practice for the task of few-shot classification.
  arXiv  Detail & Related papers  (2020-10-05T13:24:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.