Knowledge Transfer in Model-Based Reinforcement Learning Agents for Efficient Multi-Task Learning

• URL: http://arxiv.org/abs/2501.05329v1
• Date: Thu, 09 Jan 2025 15:55:08 GMT
• Title: Knowledge Transfer in Model-Based Reinforcement Learning Agents for Efficient Multi-Task Learning
• Authors: Dmytro Kuzmenko, Nadiya Shvai,
• Abstract summary: We propose an efficient knowledge transfer approach for model-based reinforcement learning.<n>We distill a high-capacity multi-task agent into a compact 1M parameter model, achieving state-of-the-art performance on the MT30 benchmark.<n>We apply FP16 post-training quantization, reducing the model size by 50% while maintaining performance.
• Score: 1.6574413179773757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose an efficient knowledge transfer approach for model-based reinforcement learning, addressing the challenge of deploying large world models in resource-constrained environments. Our method distills a high-capacity multi-task agent (317M parameters) into a compact 1M parameter model, achieving state-of-the-art performance on the MT30 benchmark with a normalized score of 28.45, a substantial improvement over the original 1M parameter model's score of 18.93. This demonstrates the ability of our distillation technique to consolidate complex multi-task knowledge effectively. Additionally, we apply FP16 post-training quantization, reducing the model size by 50% while maintaining performance. Our work bridges the gap between the power of large models and practical deployment constraints, offering a scalable solution for efficient and accessible multi-task reinforcement learning in robotics and other resource-limited domains.

Related papers

• TD-MPC-Opt: Distilling Model-Based Multi-Task Reinforcement Learning Agents [1.6574413179773757]
  We present a novel approach to knowledge transfer in model-based reinforcement learning.<n>Our method efficiently distills a high-capacity multi-task agent into a compact model.<n>Our approach addresses practical deployment limitations and offers insights into knowledge representation in large world models.
  arXiv  Detail & Related papers  (2025-07-02T15:38:49Z)
• Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer [56.898822179122476]
  We propose M3DT, a novel mixture-of-experts (MoE) framework that tackles task scalability by further unlocking the model's parameter scalability.<n> Experimental results show that, by increasing the number of experts, M3DT not only consistently enhances its performance as model expansion on the fixed task numbers, but also exhibits remarkable task scalability, successfully extending to 160 tasks with superior performance.
  arXiv  Detail & Related papers  (2025-05-30T09:08:52Z)
• MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Models [34.138699712315]
  This paper introduces a novel vision--action (VLA) model, mixture of robotic experts (MoRE) for quadruped robots. MoRE integrates multiple low-rank adaptation modules as distinct experts within a dense multi-modal large language model. Experiments demonstrate that MoRE outperforms all baselines across six different skills and exhibits superior generalization capabilities in out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-03-11T03:13:45Z)
• Efficient Multitask Learning in Small Language Models Through Upside-Down Reinforcement Learning [8.995427413172148]
  Small language models (SLMs) can achieve competitive performance in multitask prompt generation tasks. We train an SLM that achieves relevance scores within 5% of state-of-the-art models, including Llama-3, Qwen2, and Mistral, despite being up to 80 times smaller.
  arXiv  Detail & Related papers  (2025-02-14T01:39:45Z)
• Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning [59.001091197106085]
  Multi-Task Learning (MTL) for Vision Transformer aims at enhancing the model capability by tackling multiple tasks simultaneously. Most recent works have predominantly focused on designing Mixture-of-Experts (MoE) structures and in tegrating Low-Rank Adaptation (LoRA) to efficiently perform multi-task learning. We propose a novel approach dubbed Efficient Multi-Task Learning (EMTAL) by transforming a pre-trained Vision Transformer into an efficient multi-task learner.
  arXiv  Detail & Related papers  (2025-01-12T17:41:23Z)
• Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models [4.737806982257592]
  This study proposes a knowledge distillation algorithm based on large language models and feature alignment.<n>The proposed model performs very close to the state-of-the-art GPT-4 model in terms of evaluation indicators such as perplexity, BLEU, ROUGE, and CER.
  arXiv  Detail & Related papers  (2024-12-27T04:37:06Z)
• MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning [17.960185808572582]
  MoDULA is a paradigm for improved fine-tuning and parameter efficiency in multi-task learning.<n>MoDULA-Res is a new method within the MoDULA paradigm, which maintains the model's general capability by connecting universal and task-specific experts.
  arXiv  Detail & Related papers  (2024-12-10T10:55:57Z)
• On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion [23.63688816017186]
  Existing weak-to-strong methods often employ a static knowledge transfer ratio and a single small model for transferring complex knowledge. We propose a dynamic logit fusion approach that works with a series of task-specific small models, each specialized in a different task. Our method closes the performance gap by 96.4% in single-task scenarios and by 86.3% in multi-task scenarios.
  arXiv  Detail & Related papers  (2024-06-17T03:07:41Z)
• Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning [12.00246872965739]
  We propose a novel dynamic self-adaptive multiscale distillation from pre-trained multimodal large model. Our strategy employs a multiscale perspective, enabling the extraction structural knowledge across from the pre-trained multimodal large model. Our methodology streamlines pre-trained multimodal large models using only their output features and original image-level information.
  arXiv  Detail & Related papers  (2024-04-16T18:22:49Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• Parameter Efficient Multi-task Model Fusion with Partial Linearization [97.23530944186078]
  We propose a novel method to improve multi-task fusion for parameter-efficient fine-tuning techniques. Our approach partially linearizes only the adapter modules and applies task arithmetic over the linearized adapters. We demonstrate that our partial linearization technique enables a more effective fusion of multiple tasks into a single model.
  arXiv  Detail & Related papers  (2023-10-07T08:55:54Z)
• One-stop Training of Multiple Capacity Models [74.87789190840527]
  We propose a novel one-stop training framework to jointly train high-capacity and low-capactiy models. Unlike knowledge distillation, where multiple capacity models are trained from scratch separately, our approach integrates supervisions from different capacity models simultaneously.
  arXiv  Detail & Related papers  (2023-05-23T13:44:09Z)
• An Empirical Study of Multimodal Model Merging [148.48412442848795]
  Model merging is a technique that fuses multiple models trained on different tasks to generate a multi-task solution. We conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture. We propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes.
  arXiv  Detail & Related papers  (2023-04-28T15:43:21Z)
• Towards Efficient Task-Driven Model Reprogramming with Foundation Models [52.411508216448716]
  Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task.
  arXiv  Detail & Related papers  (2023-04-05T07:28:33Z)
• Scalable and Efficient MoE Training for Multitask Multilingual Models [55.987536562357086]
  We develop a system capable of scaling MoE models efficiently to trillions of parameters. We also present new training methods to improve MoE sample efficiency and leverage expert pruning strategy to improve time efficiency. A model trained with 10 billion parameters on 50 languages can achieve state-of-the-art performance in Machine Translation (MT) and multilingual natural language generation tasks.
  arXiv  Detail & Related papers  (2021-09-22T00:57:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.