Humanoid World Models: Open World Foundation Models for Humanoid Robotics

• URL: http://arxiv.org/abs/2506.01182v2
• Date: Tue, 08 Jul 2025 20:18:16 GMT
• Title: Humanoid World Models: Open World Foundation Models for Humanoid Robotics
• Abstract summary: We introduce Humanoid World Models (HWM), a family of lightweight, open-source models that forecast future egocentric video conditioned on humanoid control tokens.<n>We train two types of generative models, Masked Transformers and Flow-Matching, on 100 hours of humanoid demonstrations.<n>Our parameter-sharing techniques reduce model size by 33-53% with minimal impact on performance or visual fidelity.
• Score: 2.7781524868854715
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humanoid robots, with their human-like form, are uniquely suited for interacting in environments built for people. However, enabling humanoids to reason, plan, and act in complex open-world settings remains a challenge. World models, models that predict the future outcome of a given action, can support these capabilities by serving as a dynamics model in long-horizon planning and generating synthetic data for policy learning. We introduce Humanoid World Models (HWM), a family of lightweight, open-source models that forecast future egocentric video conditioned on humanoid control tokens. We train two types of generative models, Masked Transformers and Flow-Matching, on 100 hours of humanoid demonstrations. Additionally, we explore architectural variants with different attention mechanisms and parameter-sharing strategies. Our parameter-sharing techniques reduce model size by 33-53% with minimal impact on performance or visual fidelity. HWMs are designed to be trained and deployed in practical academic and small-lab settings, such as 1-2 GPUs.

Related papers

• GNOCHI: Generative Neural mOdel for Close Human-Human Interactions [7.759879304243429]
  This paper introduces a novel generative model for close 3D human-human interactions using a conditional variational autoencoder (cVAE)<n>We address two underlying long-standing challenges in the field of human-human interaction: data scarcity and collision awareness.<n>We demonstrate a wide variety of plausible and physically correct interactions, not possible to generate with current state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-11T17:19:32Z)
• OSCAR: Omni-Embodiment Action-Conditioned World Model for Robotics [6.134835623350618]
  We present a precise action-conditioned video world model that generalizes across different robot embodiments and enables robot policy evaluation.<n>Our model achieves significant improvement on action following, appearance quality, and motion consistency, compared to existing baselines.<n>We further deploy OSCAR to evaluate robot policies from RoboArena.
  arXiv  Detail & Related papers  (2026-06-03T05:16:41Z)
• HumanoidMimicGen: Data Generation for Loco-Manipulation via Whole-Body Planning [52.022681285103126]
  We present HumanoidMimicGen, a method for generating humanoid legged loco-manipulation data.<n>Our method adapts contact-rich whole-body skills from a handful of source demonstrations to new states.<n>We show that whole-body visuomotor policies co-trained with data generated by HumanoidMimicGen outperform those trained only on real-world data by 20%.
  arXiv  Detail & Related papers  (2026-05-26T21:57:11Z)
• Walk through Paintings: Egocentric World Models from Internet Priors [65.30611174953958]
  We present the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pretrained video diffusion model into an action-conditioned world model.<n>Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models and inject motor commands through lightweight conditioning layers.<n>Our approach scales naturally across embodiments and action spaces, ranging from 3-DoF mobile robots to 25-DoF humanoids.
  arXiv  Detail & Related papers  (2026-01-21T18:59:32Z)
• From Generated Human Videos to Physically Plausible Robot Trajectories [103.28274349461607]
  Video generation models are rapidly improving in their ability to synthesize human actions in novel contexts.<n>To realize this potential, how can a humanoid execute the human actions from generated videos in a zero-shot manner?<n>This challenge arises because generated videos are often noisy and exhibit morphological distortions that make direct imitation difficult compared to real video.<n>We propose GenMimic, a physics-aware reinforcement learning policy conditioned on 3D keypoints, and trained with symmetry regularization and keypoint-weighted tracking rewards.
  arXiv  Detail & Related papers  (2025-12-04T18:56:03Z)
• Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report [13.66563834458748]
  1X World Model Challenge introduces an open-source benchmark of real-world humanoid interaction.<n>We adapt the video generation foundation model Wan-2.2 TI2V-5B to video-state-conditioned future frame prediction.<n>Our models achieve 23.0 dB PSNR in the sampling task and a Top-500 CE of 6.6386 in the compression task, securing 1st place in both challenges.
  arXiv  Detail & Related papers  (2025-10-08T14:49:12Z)
• GWM: Towards Scalable Gaussian World Models for Robotic Manipulation [53.51622803589185]
  We propose a novel branch of world model named Gaussian World Model (GWM) for robotic manipulation.<n>At its core is a latent Diffusion Transformer (DiT) combined with a 3D variational autoencoder, enabling fine-grained scene-level future state reconstruction.<n>Both simulated and real-world experiments depict that GWM can precisely predict future scenes conditioned on diverse robot actions.
  arXiv  Detail & Related papers  (2025-08-25T02:01:09Z)
• HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers [60.86393841247567]
  HumanRAM is a novel feed-forward approach for generalizable human reconstruction and animation from monocular or sparse human images.<n>Our approach integrates human reconstruction and animation into a unified framework by introducing explicit pose conditions.<n> Experiments show that HumanRAM significantly surpasses previous methods in terms of reconstruction accuracy, animation fidelity, and generalization performance on real-world datasets.
  arXiv  Detail & Related papers  (2025-06-03T17:50:05Z)
• DreamGen: Unlocking Generalization in Robot Learning through Neural Trajectories [120.25799361925387]
  DreamGen is a pipeline for training robot policies that generalize across behaviors and environments through neural trajectories.<n>Our work establishes a promising new axis for scaling robot learning well beyond manual data collection.
  arXiv  Detail & Related papers  (2025-05-19T04:55:39Z)
• A Personalized Data-Driven Generative Model of Human Motion [0.0]
  We propose a fully data-driven approach to generate original motion that captures the unique characteristics of specific individuals.<n>Our model effectively replicates the velocity distribution and amplitude envelopes of the individual it was trained on, remaining different from other individuals.
  arXiv  Detail & Related papers  (2025-03-19T14:03:20Z)
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots [133.23509142762356]
  General-purpose robots need a versatile body and an intelligent mind.<n>Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist autonomy.<n>We introduce GR00T N1, an open foundation model for humanoid robots.
  arXiv  Detail & Related papers  (2025-03-18T21:06:21Z)
• VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation [79.00294932026266]
  VidMan is a novel framework that employs a two-stage training mechanism to enhance stability and improve data utilization efficiency. Our framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11.7% relative improvement, and demonstrates over 9% precision gains on the OXE small-scale dataset.
  arXiv  Detail & Related papers  (2024-11-14T03:13:26Z)
• Making Large Language Models into World Models with Precondition and Effect Knowledge [1.8561812622368763]
  We show that Large Language Models (LLMs) can be induced to perform two critical world model functions. We validate that the precondition and effect knowledge generated by our models aligns with human understanding of world dynamics.
  arXiv  Detail & Related papers  (2024-09-18T19:28:04Z)
• Fine-Tuning Generative Models as an Inference Method for Robotic Tasks [18.745665662647912]
  We investigate how to quickly adapt the sample generation of neural network models to observations in robotic tasks. The key idea is to quickly fine-tune the model by fitting it to generated samples matching the observed evidence. We show that our method can be applied to both autoregressive models and variational autoencoders.
  arXiv  Detail & Related papers  (2023-10-19T16:11:49Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models.<n>Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning.<n>Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)
• Real-World Humanoid Locomotion with Reinforcement Learning [92.85934954371099]
  We present a fully learning-based approach for real-world humanoid locomotion. Our controller can walk over various outdoor terrains, is robust to external disturbances, and can adapt in context.
  arXiv  Detail & Related papers  (2023-03-06T18:59:09Z)
• RT-1: Robotics Transformer for Real-World Control at Scale [98.09428483862165]
  We present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks.
  arXiv  Detail & Related papers  (2022-12-13T18:55:15Z)
• Factored World Models for Zero-Shot Generalization in Robotic Manipulation [7.258229016768018]
  We learn to generalize over robotic pick-and-place tasks using object-factored world models. We use a residual stack of graph neural networks that receive action information at multiple levels in both their node and edge neural networks. We show that an ensemble of our models can be used to plan for tasks involving up to 12 pick and place actions using search.
  arXiv  Detail & Related papers  (2022-02-10T21:26:11Z)
• Real-time Pose Estimation from Images for Multiple Humanoid Robots [45.182157261640675]
  We present a lightweight pose estimation model that can work in real-time on humanoid robots in the RoboCup Humanoid League environment. The results of this work have the potential to enable many advanced behaviors for soccer-playing robots.
  arXiv  Detail & Related papers  (2021-07-06T15:33:57Z)
• S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling [103.65625425020129]
  We represent the pedestrian's shape, pose and skinning weights as neural implicit functions that are directly learned from data. We demonstrate the effectiveness of our approach on various datasets and show that our reconstructions outperform existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-01-17T02:16:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.