Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks

• URL: http://arxiv.org/abs/2503.01885v1
• Date: Wed, 26 Feb 2025 22:45:25 GMT
• Title: Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks
• Authors: Luise Ge, Michael Lanier, Anindya Sarkar, Bengisu Guresti, Yevgeniy Vorobeychik, Chongjie Zhang,
• Abstract summary: We propose a novel approach for learning a policy committee that includes at least one near-optimal policy with high probability for tasks encountered during execution.<n>Our experiments on MuJoCo and Meta-World show that the proposed approach outperforms state-of-the-art multi-task, meta-, and task clustering baselines in training, generalization, and few-shot learning.
• Score: 40.2989900672992
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many dynamic decision problems, such as robotic control, involve a series of tasks, many of which are unknown at training time. Typical approaches for these problems, such as multi-task and meta reinforcement learning, do not generalize well when the tasks are diverse. On the other hand, approaches that aim to tackle task diversity, such as using task embedding as policy context and task clustering, typically lack performance guarantees and require a large number of training tasks. To address these challenges, we propose a novel approach for learning a policy committee that includes at least one near-optimal policy with high probability for tasks encountered during execution. While we show that this problem is in general inapproximable, we present two practical algorithmic solutions. The first yields provable approximation and task sample complexity guarantees when tasks are low-dimensional (the best we can do due to inapproximability), whereas the second is a general and practical gradient-based approach. In addition, we provide a provable sample complexity bound for few-shot learning. Our experiments on MuJoCo and Meta-World show that the proposed approach outperforms state-of-the-art multi-task, meta-, and task clustering baselines in training, generalization, and few-shot learning, often by a large margin.

Related papers

• Active Fine-Tuning of Generalist Policies [54.65568433408307]
  We propose AMF (Active Multi-task Fine-tuning) to maximize multi-task policy performance under a limited demonstration budget. We derive performance guarantees for AMF under regularity assumptions and demonstrate its empirical effectiveness in complex and high-dimensional environments.
  arXiv  Detail & Related papers  (2024-10-07T13:26:36Z)
• Learning Complex Teamwork Tasks Using a Given Sub-task Decomposition [11.998708550268978]
  We propose an approach which uses an expert-provided decomposition of a task into simpler multi-agent sub-tasks. In each sub-task, a subset of the entire team is trained to acquire sub-task-specific policies. The sub-teams are then merged and transferred to the target task, where their policies are collectively fine-tuned to solve the more complex target task.
  arXiv  Detail & Related papers  (2023-02-09T21:24:56Z)
• Robust Subtask Learning for Compositional Generalization [20.54144051436337]
  We focus on the problem of training subtask policies in a way that they can be used to perform any task. We aim to maximize the worst-case performance over all tasks as opposed to the average-case performance.
  arXiv  Detail & Related papers  (2023-02-06T18:19:25Z)
• Leveraging convergence behavior to balance conflicting tasks in multi-task learning [3.6212652499950138]
  Multi-Task Learning uses correlated tasks to improve performance generalization. Tasks often conflict with each other, which makes it challenging to define how the gradients of multiple tasks should be combined. We propose a method that takes into account temporal behaviour of the gradients to create a dynamic bias that adjust the importance of each task during the backpropagation.
  arXiv  Detail & Related papers  (2022-04-14T01:52:34Z)
• Variational Multi-Task Learning with Gumbel-Softmax Priors [105.22406384964144]
  Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
  arXiv  Detail & Related papers  (2021-11-09T18:49:45Z)
• Multi-Task Learning with Sequence-Conditioned Transporter Networks [67.57293592529517]
  We aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. We propose a new suite of benchmark aimed at compositional tasks, MultiRavens, which allows defining custom task combinations. Second, we propose a vision-based end-to-end system architecture, Sequence-Conditioned Transporter Networks, which augments Goal-Conditioned Transporter Networks with sequence-conditioning and weighted sampling.
  arXiv  Detail & Related papers  (2021-09-15T21:19:11Z)
• Efficiently Identifying Task Groupings for Multi-Task Learning [55.80489920205404]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We suggest an approach to select which tasks should train together in multi-task learning models. Our method determines task groupings in a single training run by co-training all tasks together and quantifying the effect to which one task's gradient would affect another task's loss.
  arXiv  Detail & Related papers  (2021-09-10T02:01:43Z)
• Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization [100.72335252255989]
  We study the problem of learning exploration-exploitation strategies that effectively adapt to dynamic environments. We propose a novel algorithm that regularizes the training of an RNN-based policy using informed policies trained to maximize the reward in each task.
  arXiv  Detail & Related papers  (2020-05-06T16:14:48Z)
• A Simple General Approach to Balance Task Difficulty in Multi-Task Learning [4.531240717484252]
  In multi-task learning, difficulty levels of different tasks are varying. We propose a Balanced Multi-Task Learning (BMTL) framework. The proposed BMTL framework is very simple and it can be combined with most multi-task learning models.
  arXiv  Detail & Related papers  (2020-02-12T04:31:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.