MetaWorld-X: Hierarchical World Modeling via VLM-Orchestrated Experts for Humanoid Loco-Manipulation

• URL: http://arxiv.org/abs/2603.08572v1
• Date: Mon, 09 Mar 2026 16:28:26 GMT
• Title: MetaWorld-X: Hierarchical World Modeling via VLM-Orchestrated Experts for Humanoid Loco-Manipulation
• Abstract summary: Existing reinforcement learning approaches rely on a single monolithic policy to acquire multiple skills.<n>We propose MetaWorld-X, a hierarchical world model framework for humanoid control.<n>Our method decomposes complex control problems into a set of specialized expert policies.
• Score: 7.721920759084426
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning natural, stable, and compositionally generalizable whole-body control policies for humanoid robots performing simultaneous locomotion and manipulation (loco-manipulation) remains a fundamental challenge in robotics. Existing reinforcement learning approaches typically rely on a single monolithic policy to acquire multiple skills, which often leads to cross-skill gradient interference and motion pattern conflicts in high-degree-of-freedom systems. As a result, generated behaviors frequently exhibit unnatural movements, limited stability, and poor generalization to complex task compositions. To address these limitations, we propose MetaWorld-X, a hierarchical world model framework for humanoid control. Guided by a divide-and-conquer principle, our method decomposes complex control problems into a set of specialized expert policies (Specialized Expert Policies, SEP). Each expert is trained under human motion priors through imitation-constrained reinforcement learning, introducing biomechanically consistent inductive biases that ensure natural and physically plausible motion generation. Building upon this foundation, we further develop an Intelligent Routing Mechanism (IRM) supervised by a Vision-Language Model (VLM), enabling semantic-driven expert composition. The VLM-guided router dynamically integrates expert policies according to high-level task semantics, facilitating compositional generalization and adaptive execution in multi-stage loco-manipulation tasks.

Related papers

• LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation [16.259708213720383]
  Long-horizon humanoid loco-manipulation requires versatile whole-body skills and reliable high-level decision making.<n>Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies.<n>We propose textbfLUCID, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model.
  arXiv  Detail & Related papers  (2026-08-07T20:26:34Z)
• Extreme-RGMT: Continual Learning of Highly Dynamic Skills for Robust Generalist Humanoid Control [12.247654049983522]
  We introduce Extreme-RGMT, a two-stage continual learning framework for robust generalist humanoid control.<n>Extreme-RGMT learns a generalist motion-tracking base policy from diverse multi-source motion data, then employs an asymmetric skill acquisition and capability consolidation mechanism.<n>Experiments show that Extreme-RGMT achieves state-of-the-art generalist whole-body motion-tracking performance.
  arXiv  Detail & Related papers  (2026-07-22T13:10:22Z)
• ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation [55.467742403416175]
  We introduce a physics-driven neural algorithm that translates large-scale motion capture to humanoid embodiments.<n>We learn a unified multimodal controller that supports both dense references and sparse task specifications.<n>Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from egocentric perception.
  arXiv  Detail & Related papers  (2026-03-03T18:59:29Z)
• InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions [58.329946838699044]
  Humans rarely plan whole-body interactions with objects at the level of explicit whole-body movements.<n>Scaling such priors is key to enabling humanoids to compose and generalize loco-manipulation skills.<n>We introduce InterPrior, a framework that learns a unified generative controller through large-scale imitation pretraining and post-training by reinforcement learning.
  arXiv  Detail & Related papers  (2026-02-05T18:59:27Z)
• PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes [18.916978556216097]
  PILOT is a single-stage reinforcement learning framework tailored for perceptive loco-manipulation.<n>It synergizes perceptive locomotion and expansive whole-body control within a single policy.<n> PILOT demonstrates superior stability, command tracking precision, and terrain traversability compared to existing baselines.
  arXiv  Detail & Related papers  (2026-01-24T12:25:20Z)
• Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation [90.90219129619344]
  This paper presents a novel R-prior-S, Recurrent Geometric-priormodal Policy with Spiking features.<n>To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases.<n>For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network.
  arXiv  Detail & Related papers  (2026-01-13T23:36:30Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• KungfuBot2: Learning Versatile Motion Skills for Humanoid Whole-Body Control [30.738592041595933]
  We present VMS, a unified whole-body controller that enables humanoid robots to learn diverse and dynamic behaviors within a single policy.<n>Our framework integrates a hybrid tracking objective that balances local motion fidelity with global trajectory consistency.<n>We validate VMS specialization extensively in both simulation and real-world experiments, demonstrating accurate imitation of dynamic skills, stable performance over minute-long sequences, and strong generalization to unseen motions.
  arXiv  Detail & Related papers  (2025-09-20T11:31:14Z)
• From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robots [35.26305396688982]
  BumbleBee is an expert-generalist learning framework that combines motion clustering and sim-to-real adaptation.<n> Experiments on two simulations and a real humanoid robot demonstrate that BB achieves state-of-the-art general whole-body control.
  arXiv  Detail & Related papers  (2025-06-15T09:09:34Z)
• Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids [56.892520712892804]
  We introduce a practical sim-to-real RL recipe that trains a humanoid robot to perform three dexterous manipulation tasks.<n>We demonstrate high success rates on unseen objects and robust, adaptive policy behaviors.
  arXiv  Detail & Related papers  (2025-02-27T18:59:52Z)
• COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Grasping [56.907940167333656]
  Occluded robot grasping is where the desired grasp poses are kinematically infeasible due to environmental constraints such as surface collisions.<n>Traditional robot manipulation approaches struggle with the complexity of non-prehensile or bimanual strategies commonly used by humans.<n>We introduce Constraint-based Manipulation for Bimanual Occluded Grasping (COMBO-Grasp), a learning-based approach which leverages two coordinated policies.
  arXiv  Detail & Related papers  (2025-02-12T01:31:01Z)
• Towards Natural Language-Driven Assembly Using Foundation Models [11.710022685486914]
  Large Language Models (LLMs) and strong vision models have enabled rapid research and development in the field of Vision-Language-Action models. We present a global control policy based on LLMs that can transfer the control policy to a finite set of skills that are specifically trained to perform high-precision tasks. The integration of LLMs into this framework underscores their significance in not only interpreting and processing language inputs but also in enriching the control mechanisms for diverse and intricate robotic operations.
  arXiv  Detail & Related papers  (2024-06-23T12:14:37Z)
• On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning [32.070068456106895]
  We apply model-free Deep Reinforcement Learning for training a general and robust humanoid push-recovery policy in a simulation environment. Our method targets high-dimensional whole-body humanoid control and is validated on the iCub humanoid.
  arXiv  Detail & Related papers  (2021-04-29T17:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.