Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learning

• URL: http://arxiv.org/abs/2408.13567v1
• Date: Sat, 24 Aug 2024 12:37:03 GMT
• Title: Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learning
• Authors: Mingliang Zhang, Sichang Su, Chengyang He, Guillaume Sartoretti,
• Abstract summary: HyGen is a novel hybrid MARL framework, which integrates online and offline learning to ensure both multi-task generalization and training efficiency. We empirically demonstrate that our framework effectively extracts and refines general skills, yielding impressive generalization to unseen tasks.
• Score: 7.6201940008534175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In multi-agent reinforcement learning (MARL), achieving multi-task generalization to diverse agents and objectives presents significant challenges. Existing online MARL algorithms primarily focus on single-task performance, but their lack of multi-task generalization capabilities typically results in substantial computational waste and limited real-life applicability. Meanwhile, existing offline multi-task MARL approaches are heavily dependent on data quality, often resulting in poor performance on unseen tasks. In this paper, we introduce HyGen, a novel hybrid MARL framework, Hybrid Training for Enhanced Multi-Task Generalization, which integrates online and offline learning to ensure both multi-task generalization and training efficiency. Specifically, our framework extracts potential general skills from offline multi-task datasets. We then train policies to select the optimal skills under the centralized training and decentralized execution paradigm (CTDE). During this stage, we utilize a replay buffer that integrates both offline data and online interactions. We empirically demonstrate that our framework effectively extracts and refines general skills, yielding impressive generalization to unseen tasks. Comparative analyses on the StarCraft multi-agent challenge show that HyGen outperforms a wide range of existing solely online and offline methods.

Related papers

• Omni-Thinker: Scaling Cross-Domain Generalization in LLMs via Multi-Task RL with Hybrid Rewards [50.21528417884747]
  We introduce Omni-Thinker, a unified reinforcement learning framework that enhances large language models (LLMs) performance across diverse tasks.<n>Our approach enables consistent optimization across task types and scales RL-based training to subjective domains.<n> Experimental results across four domains reveal that curriculum learning improves performance by 5.2% over joint training and 9.1% over model merging.
  arXiv  Detail & Related papers  (2025-07-20T01:50:16Z)
• Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration [63.90193684394165]
  We introduce multi-agent cross-task experiential learning (MAEL), a novel framework that endows LLM-driven agents with explicit cross-task learning and experience accumulation.<n>During the experiential learning phase, we quantify the quality for each step in the task-solving workflow and store the resulting rewards.<n>During inference, agents retrieve high-reward, task-relevant experiences as few-shot examples to enhance the effectiveness of each reasoning step.
  arXiv  Detail & Related papers  (2025-05-29T07:24:37Z)
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners [60.75160178669076]
  We show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online reinforcement learning.<n>We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL.
  arXiv  Detail & Related papers  (2025-05-29T06:41:45Z)
• Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation [16.107745320103444]
  Hierarchical and Separate Skill Discovery (HiSSD) is a novel approach for generalizable offline multi-task MARL through skill learning. HiSSD leverages a hierarchical framework that jointly learns common and task-specific skills. The task-specific skills represent the priors of each task and achieve a task-guided fine-grained action execution.
  arXiv  Detail & Related papers  (2025-03-27T06:35:59Z)
• Multi-task Representation Learning for Mixed Integer Linear Programming [13.106799330951842]
  This paper introduces the first multi-task learning framework for ML-guided MILP solving. We demonstrate that our multi-task learning model performs similarly to specialized models within the same distribution. It significantly outperforms them in generalization across problem sizes and tasks.
  arXiv  Detail & Related papers  (2024-12-18T23:33:32Z)
• Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning [11.790581500542439]
  Reinforcement learning (RL) with diverse offline datasets can have the advantage of leveraging the relation of multiple tasks. We present a skill-based multi-task RL technique on heterogeneous datasets that are generated by behavior policies of different quality. We show that our multi-task offline RL approach is robust to the mixed configurations of different-quality datasets.
  arXiv  Detail & Related papers  (2024-08-28T07:36:20Z)
• Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks [53.44714413181162]
  This paper shows that when an agent is trained on a sufficiently diverse set of tasks, a generic policy-sharing algorithm with myopic exploration design can be sample-efficient. To the best of our knowledge, this is the first theoretical demonstration of the "exploration benefits" of MTRL.
  arXiv  Detail & Related papers  (2024-03-03T22:57:44Z)
• Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization [23.416448404647305]
  OMIGA is a new offline m ulti-agent RL algorithm with implicit global-to-local v alue regularization. We show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks.
  arXiv  Detail & Related papers  (2023-07-21T14:37:54Z)
• Multi-task Hierarchical Adversarial Inverse Reinforcement Learning [40.60364143826424]
  Multi-task Imitation Learning (MIL) aims to train a policy capable of performing a distribution of tasks based on multi-task expert demonstrations. Existing MIL algorithms suffer from low data efficiency and poor performance on complex long-horizontal tasks. We develop Multi-task Hierarchical Adversarial Inverse Reinforcement Learning (MH-AIRL) to learn hierarchically-structured multi-task policies.
  arXiv  Detail & Related papers  (2023-05-22T01:58:40Z)
• Learning From Good Trajectories in Offline Multi-Agent Reinforcement Learning [98.07495732562654]
  offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets. One agent learned by offline MARL often inherits this random policy, jeopardizing the performance of the entire team. We propose a novel framework called Shared Individual Trajectories (SIT) to address this problem.
  arXiv  Detail & Related papers  (2022-11-28T18:11:26Z)
• Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving [103.745551954983]
  In this paper, we investigate the transfer performance of various types of self-supervised methods, including MoCo and SimCLR, on three downstream tasks. We find that their performances are sub-optimal or even lag far behind the single-task baseline. We propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training.
  arXiv  Detail & Related papers  (2022-09-19T12:15:31Z)
• Meta Reinforcement Learning with Successor Feature Based Context [51.35452583759734]
  We propose a novel meta-RL approach that achieves competitive performance comparing to existing meta-RL algorithms. Our method does not only learn high-quality policies for multiple tasks simultaneously but also can quickly adapt to new tasks with a small amount of training.
  arXiv  Detail & Related papers  (2022-07-29T14:52:47Z)
• Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Conquers All StarCraftII Tasks [43.588686040547486]
  offline pre-training with online fine-tuning has never been studied, nor datasets or benchmarks for offline MARL research are available. We propose the novel architecture of multi-agent decision transformer (MADT) for effective offline learning. When evaluated on StarCraft II offline dataset, MADT demonstrates superior performance than state-of-the-art offline RL baselines.
  arXiv  Detail & Related papers  (2021-12-06T08:11:05Z)
• Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction [125.18248926508045]
  We propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for both multimodal fusion and multitask learning. CEN dynamically exchanges channels betweenworks of different modalities. For the application of dense image prediction, the validity of CEN is tested by four different scenarios.
  arXiv  Detail & Related papers  (2021-12-04T05:47:54Z)
• MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning [61.28547338576706]
  Population-based multi-agent reinforcement learning (PB-MARL) refers to the series of methods nested with reinforcement learning (RL) algorithms. We present MALib, a scalable and efficient computing framework for PB-MARL.
  arXiv  Detail & Related papers  (2021-06-05T03:27:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.