Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control

• URL: http://arxiv.org/abs/2412.12147v1
• Date: Tue, 10 Dec 2024 19:10:17 GMT
• Title: Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control
• Abstract summary: Generalizing across robot embodiments and tasks is crucial for adaptive robotic systems.<n>We introduce a few-shot behavior cloning framework to simultaneously generalize to unseen embodiments and tasks.<n>A matching-based policy network then predicts actions from a few demonstrations, producing an adaptive policy that is robust to over-fitting.
• Score: 19.259973997622346
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Generalizing across robot embodiments and tasks is crucial for adaptive robotic systems. Modular policy learning approaches adapt to new embodiments but are limited to specific tasks, while few-shot imitation learning (IL) approaches often focus on a single embodiment. In this paper, we introduce a few-shot behavior cloning framework to simultaneously generalize to unseen embodiments and tasks using a few (\emph{e.g.,} five) reward-free demonstrations. Our framework leverages a joint-level input-output representation to unify the state and action spaces of heterogeneous embodiments and employs a novel structure-motion state encoder that is parameterized to capture both shared knowledge across all embodiments and embodiment-specific knowledge. A matching-based policy network then predicts actions from a few demonstrations, producing an adaptive policy that is robust to over-fitting. Evaluated in the DeepMind Control suite, our framework termed \modelname{} demonstrates superior few-shot generalization to unseen embodiments and tasks over modular policy learning and few-shot IL approaches. Codes are available at \href{https://github.com/SeongwoongCho/meta-controller}{https://github.com/SeongwoongCho/meta-controller}.

Related papers

• DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation [18.601805553878638]
  DyPES-VLA is a cross-embodiment VLA that learns shared Dynamics Priors and Embodiment-Specific control.<n>As a generalist policy, our ourmethod achieves state-of-the-art performance across simulation and real-world evaluations.
  arXiv  Detail & Related papers  (2026-08-06T17:59:20Z)
• KITE: Decoupling Kinematics and Interaction for Zero-Shot Cross-Embodiment Manipulation [2.532260218883922]
  Generalizing manipulation policies across robot embodiments remains difficult because standard policies entangle task reasoning with embodiment-specific motor control.<n>We introduce Kinematic Interaction Transfer across Embodiments (KITE), which decouples manipulation into embodiment-agnostic task reasoning and embodiment-specific motor control.<n>KITE consistently achieves zero-shot transfer to structurally different target embodiments, outperforming state-of-the-art baselines in transfer success and task-embodiment scope.
  arXiv  Detail & Related papers  (2026-06-20T15:51:30Z)
• AdaTracker: Learning Adaptive In-Context Policy for Cross-Embodiment Active Visual Tracking [42.310377711916104]
  Existing approaches typically train separate models for each embodiment, leading to poor scalability and limited generalization.<n>AdaTracker is an adaptive in-context policy learning framework that robustly tracks targets on diverse robot morphologies.<n> Experiments in both simulation and the real world demonstrate that AdaTracker significantly outperforms state-of-the-art methods in cross-embodiment generalization, sample efficiency, and zero-shot adaptation.
  arXiv  Detail & Related papers  (2026-04-22T08:06:32Z)
• Model Evolution Framework with Genetic Algorithm for Multi-Task Reinforcement Learning [85.91908329457081]
  Multi-task reinforcement learning employs a single policy to complete various tasks, aiming to develop an agent with generalizability across different scenarios. Existing approaches typically use a routing network to generate specific routes for each task and reconstruct a set of modules into diverse models to complete multiple tasks simultaneously. We propose a Model Evolution framework with Genetic Algorithm (MEGA), which enables the model to evolve during training according to the difficulty of the tasks.
  arXiv  Detail & Related papers  (2025-02-19T09:22:34Z)
• Towards Modality Generalization: A Benchmark and Prospective Analysis [56.84045461854789]
  This paper introduces Modality Generalization (MG), which focuses on enabling models to generalize to unseen modalities. We propose a comprehensive benchmark featuring multi-modal algorithms and adapt existing methods that focus on generalization. Our work provides a foundation for advancing robust and adaptable multi-modal models, enabling them to handle unseen modalities in realistic scenarios.
  arXiv  Detail & Related papers  (2024-12-24T08:38:35Z)
• Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion [41.52811286996212]
  Make-An-Agent is a novel policy parameter generator for behavior-to-policy generation. We show how it can generate a control policy for an agent using just one demonstration of desired behaviors as a prompt. We also deploy policies generated by Make-An-Agent onto real-world robots on locomotion tasks.
  arXiv  Detail & Related papers  (2024-07-15T17:59:57Z)
• Dreamitate: Real-World Visuomotor Policy Learning via Video Generation [49.03287909942888]
  We propose a visuomotor policy learning framework that fine-tunes a video diffusion model on human demonstrations of a given task. We generate an example of an execution of the task conditioned on images of a novel scene, and use this synthesized execution directly to control the robot.
  arXiv  Detail & Related papers  (2024-06-24T17:59:45Z)
• Code Models are Zero-shot Precondition Reasoners [83.8561159080672]
  We use code representations to reason about action preconditions for sequential decision making tasks. We propose a precondition-aware action sampling strategy that ensures actions predicted by a policy are consistent with preconditions.
  arXiv  Detail & Related papers  (2023-11-16T06:19:27Z)
• Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments [45.213059639254475]
  We propose a new topic called imitator learning (ItorL) It aims to derive an imitator module that can reconstruct the imitation policies based on very limited expert demonstrations. For autonomous imitation policy building, we design a demonstration-based attention architecture for imitator policy.
  arXiv  Detail & Related papers  (2023-10-09T13:35:28Z)
• Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation [45.312333134810665]
  State-of-the-art methods to tackle few-shot imitation rely on meta-learning. Recent work has shown that fine-tuners outperform meta-learners in few-shot image classification tasks. We release an open source dataset called iMuJoCo consisting of 154 variants of popular OpenAI-Gym MuJoCo environments.
  arXiv  Detail & Related papers  (2023-06-23T15:29:15Z)
• Composite Motion Learning with Task Control [0.6882042556551609]
  We present a deep learning method for composite and task-driven motion control for physically simulated characters. We learn decoupled motions for specific body parts from multiple reference motions simultaneously and directly by leveraging the use of multiple discriminators in a GAN-like setup. We show the applicability of our approach on a variety of challenging multi-objective tasks involving both composite motion imitation and multiple goal-directed control.
  arXiv  Detail & Related papers  (2023-05-05T05:02:41Z)
• Inferring Versatile Behavior from Demonstrations by Matching Geometric Descriptors [72.62423312645953]
  Humans intuitively solve tasks in versatile ways, varying their behavior in terms of trajectory-based planning and for individual steps. Current Imitation Learning algorithms often only consider unimodal expert demonstrations and act in a state-action-based setting. Instead, we combine a mixture of movement primitives with a distribution matching objective to learn versatile behaviors that match the expert's behavior and versatility.
  arXiv  Detail & Related papers  (2022-10-17T16:42:59Z)
• Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs [63.936622239286685]
  We find that interference among different tasks and modalities is the main factor to this phenomenon. We introduce the Conditional Mixture-of-Experts (Conditional MoEs) to generalist models. Code and pre-trained generalist models shall be released.
  arXiv  Detail & Related papers  (2022-06-09T17:59:59Z)
• Learning Multi-Objective Curricula for Deep Reinforcement Learning [55.27879754113767]
  Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL) In this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula. In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum.
  arXiv  Detail & Related papers  (2021-10-06T19:30:25Z)
• Imitation Learning from MPC for Quadrupedal Multi-Gait Control [63.617157490920505]
  We present a learning algorithm for training a single policy that imitates multiple gaits of a walking robot. We use and extend MPC-Net, which is an Imitation Learning approach guided by Model Predictive Control. We validate our approach on hardware and show that a single learned policy can replace its teacher to control multiple gaits.
  arXiv  Detail & Related papers  (2021-03-26T08:48:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.