DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusion

• URL: http://arxiv.org/abs/2509.14353v3
• Date: Tue, 30 Sep 2025 15:48:45 GMT
• Title: DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusion
• Abstract summary: We introduce DreamControl, a novel methodology for learning autonomous whole-body humanoid skills.<n>Our core innovation is the use of a diffusion prior trained on human motion data, which guides an RL policy in simulation to complete specific tasks of interest.<n>We validate DreamControl's effectiveness on a Unitree G1 robot across a diverse set of challenging tasks involving simultaneous lower and upper body control and object interaction.
• Score: 21.619868038556987
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce DreamControl, a novel methodology for learning autonomous whole-body humanoid skills. DreamControl leverages the strengths of diffusion models and Reinforcement Learning (RL): our core innovation is the use of a diffusion prior trained on human motion data, which subsequently guides an RL policy in simulation to complete specific tasks of interest (e.g., opening a drawer or picking up an object). We demonstrate that this human motion-informed prior allows RL to discover solutions unattainable by direct RL, and that diffusion models inherently promote natural looking motions, aiding in sim-to-real transfer. We validate DreamControl's effectiveness on a Unitree G1 robot across a diverse set of challenging tasks involving simultaneous lower and upper body control and object interaction. Project website at https://genrobo.github.io/DreamControl/

Related papers

• DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priors [20.558671221723422]
  We propose an improved framework that trains a guided diffusion model directly in the humanoid robot's motion space.<n>We demonstrate that our approach captures a wider range of skills due to the larger training data mixture.<n>We show that scaling the generation of reference trajectories is important for achieving robust downstream RL policies.
  arXiv  Detail & Related papers  (2026-03-31T20:11:49Z)
• ZeroWBC: Learning Natural Visuomotor Humanoid Control Directly from Human Egocentric Video [52.78703020909145]
  We introduce ZeroWBC, a novel framework that learns a natural humanoid visuomotor control policy directly from human egocentric videos.<n>Our approach first fine-tunes a Vision-Language Model (VLM) to predict future whole-body human motions based on text instructions and egocentric visual context.<n>Experiments on the Unitree G1 humanoid robot demonstrate that our method outperforms baseline approaches in motion naturalness and versatility.
  arXiv  Detail & Related papers  (2026-03-10T04:19:43Z)
• World Action Models are Zero-shot Policies [111.91938055103633]
  We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone.<n>By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data.<n>We demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance.
  arXiv  Detail & Related papers  (2026-02-17T15:04:02Z)
• Learning Sim-to-Real Humanoid Locomotion in 15 Minutes [51.500643119683225]
  We introduce a simple and practical recipe based on off-policy RL algorithms, i.e., FastSAC and FastTD3.<n>Our simple recipe stabilizes off-policy RL algorithms at massive scale with thousands of parallel environments.<n>We demonstrate rapid end-to-end learning of humanoid locomotion controllers on Unitree G1 and Booster T1 robots.
  arXiv  Detail & Related papers  (2025-12-01T18:55:17Z)
• Feel the Force: Contact-Driven Learning from Humans [52.36160086934298]
  Controlling fine-grained forces during manipulation remains a core challenge in robotics.<n>We present FeelTheForce, a robot learning system that models human tactile behavior to learn force-sensitive manipulation.<n>Our approach grounds robust low-level force control in scalable human supervision, achieving a 77% success rate across 5 force-sensitive manipulation tasks.
  arXiv  Detail & Related papers  (2025-06-02T17:57:52Z)
• Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach [55.76249793590689]
  Video-Enhanced Offline RL (VeoRL) is a model-based method that constructs an interactive world model from diverse, unlabeled video data readily available online.<n>VeoRL achieves substantial performance gains across visual control tasks in robotic manipulation, autonomous driving, and open-world video games.
  arXiv  Detail & Related papers  (2025-05-10T00:54:12Z)
• TWIST: Teleoperated Whole-Body Imitation System [28.597388162969057]
  We present the Teleoperated Whole-Body Imitation System (TWIST), a system for humanoid teleoperation through whole-body motion imitation.<n>We develop a robust, adaptive, and responsive whole-body controller using a combination of reinforcement learning and behavior cloning.<n>TWIST enables real-world humanoid robots to achieve unprecedented, versatile, and coordinated whole-body motor skills.
  arXiv  Detail & Related papers  (2025-05-05T17:59:03Z)
• A Generative System for Robot-to-Human Handovers: from Intent Inference to Spatial Configuration Imagery [0.0]
  We propose a novel system for robot-to-human object handover that emulates human coworker interactions.<n>Our system focus on 1. inferring human handover intents, 2. imagining spatial handover configuration.<n> Experimental results demonstrate that our approach effectively interprets human cues and achieves fluent, human-like handovers.
  arXiv  Detail & Related papers  (2025-03-05T15:13:54Z)
• Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control [106.32794844077534]
  This paper presents a study on using deep reinforcement learning to create dynamic locomotion controllers for bipedal robots. We develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. This work pushes the limits of agility for bipedal robots through extensive real-world experiments.
  arXiv  Detail & Related papers  (2024-01-30T10:48:43Z)
• InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint [67.6297384588837]
  We introduce a novel controllable motion generation method, InterControl, to encourage the synthesized motions maintaining the desired distance between joint pairs. We demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model.
  arXiv  Detail & Related papers  (2023-11-27T14:32:33Z)
• Universal Humanoid Motion Representations for Physics-Based Control [71.46142106079292]
  We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. We first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator.
  arXiv  Detail & Related papers  (2023-10-06T20:48:43Z)
• Decoupling Skill Learning from Robotic Control for Generalizable Object Manipulation [35.34044822433743]
  Recent works in robotic manipulation have shown potential for tackling a range of tasks. We conjecture that this is due to the high-dimensional action space for joint control. In this paper, we take an alternative approach and separate the task of learning 'what to do' from 'how to do it' The whole-body robotic kinematic control is optimized to execute the high-dimensional joint motion to reach the goals in the workspace.
  arXiv  Detail & Related papers  (2023-03-07T16:31:13Z)
• Human-to-Robot Imitation in the Wild [50.49660984318492]
  We propose an efficient one-shot robot learning algorithm, centered around learning from a third-person perspective. We show one-shot generalization and success in real-world settings, including 20 different manipulation tasks in the wild.
  arXiv  Detail & Related papers  (2022-07-19T17:59:59Z)
• Human-AI Shared Control via Frequency-based Policy Dissection [34.0399894373716]
  Human-AI shared control allows human to interact and collaborate with AI to accomplish control tasks in complex environments. Previous Reinforcement Learning (RL) methods attempt the goal-conditioned design to achieve human-controllable policies. We develop a simple yet effective frequency-based approach called textitPolicy Dissection to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior.
  arXiv  Detail & Related papers  (2022-05-31T23:57:55Z)
• UniCon: Universal Neural Controller For Physics-based Character Motion [70.45421551688332]
  We propose a physics-based universal neural controller (UniCon) that learns to master thousands of motions with different styles by learning on large-scale motion datasets. UniCon can support keyboard-driven control, compose motion sequences drawn from a large pool of locomotion and acrobatics skills and teleport a person captured on video to a physics-based virtual avatar.
  arXiv  Detail & Related papers  (2020-11-30T18:51:16Z)
• AirCapRL: Autonomous Aerial Human Motion Capture using Deep Reinforcement Learning [38.429105809093116]
  We introduce a deep reinforcement learning (RL) based multi-robot formation controller for the task of autonomous aerial human motion capture (MoCap) We focus on vision-based MoCap, where the objective is to estimate the trajectory of body pose and shape a single moving person using multiple aerial vehicles.
  arXiv  Detail & Related papers  (2020-07-13T12:30:31Z)
• Learning to Walk in the Real World with Minimal Human Effort [80.7342153519654]
  We develop a system for learning legged locomotion policies with deep RL in the real world with minimal human effort. Our system can automatically and efficiently learn locomotion skills on a Minitaur robot with little human intervention.
  arXiv  Detail & Related papers  (2020-02-20T03:36:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.