Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skills

• URL: http://arxiv.org/abs/2503.12533v1
• Date: Sun, 16 Mar 2025 14:53:53 GMT
• Title: Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skills
• Authors: Haoqi Yuan, Yu Bai, Yuhui Fu, Bohan Zhou, Yicheng Feng, Xinrun Xu, Yi Zhan, Börje F. Karlsson, Zongqing Lu,
• Abstract summary: Building autonomous robotic agents capable of achieving human-level performance in real-world embodied tasks is an ultimate goal in humanoid robot research.<n>Recent advances have made significant progress in high-level cognition with Foundation Models (FMs) and low-level skill development for humanoid robots.<n>We introduce Being-0, a hierarchical agent framework that integrates an FM with a modular skill library.<n>Being-0 achieves efficient, real-time performance on a full-sized humanoid robot equipped with dexterous hands and active vision.
• Score: 31.788094786664324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Building autonomous robotic agents capable of achieving human-level performance in real-world embodied tasks is an ultimate goal in humanoid robot research. Recent advances have made significant progress in high-level cognition with Foundation Models (FMs) and low-level skill development for humanoid robots. However, directly combining these components often results in poor robustness and efficiency due to compounding errors in long-horizon tasks and the varied latency of different modules. We introduce Being-0, a hierarchical agent framework that integrates an FM with a modular skill library. The FM handles high-level cognitive tasks such as instruction understanding, task planning, and reasoning, while the skill library provides stable locomotion and dexterous manipulation for low-level control. To bridge the gap between these levels, we propose a novel Connector module, powered by a lightweight vision-language model (VLM). The Connector enhances the FM's embodied capabilities by translating language-based plans into actionable skill commands and dynamically coordinating locomotion and manipulation to improve task success. With all components, except the FM, deployable on low-cost onboard computation devices, Being-0 achieves efficient, real-time performance on a full-sized humanoid robot equipped with dexterous hands and active vision. Extensive experiments in large indoor environments demonstrate Being-0's effectiveness in solving complex, long-horizon tasks that require challenging navigation and manipulation subtasks. For further details and videos, visit https://beingbeyond.github.io/being-0.

Related papers

• Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models [49.4824734958566]
  Chain-of-Modality (CoM) enables Vision Language Models to reason about multimodal human demonstration data. CoM refines a task plan and generates detailed control parameters, enabling robots to perform manipulation tasks based on a single multimodal human video prompt.
  arXiv  Detail & Related papers  (2025-04-17T21:31:23Z)
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots [133.23509142762356]
  General-purpose robots need a versatile body and an intelligent mind. Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist autonomy. We introduce GR00T N1, an open foundation model for humanoid robots.
  arXiv  Detail & Related papers  (2025-03-18T21:06:21Z)
• HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit [52.12750762494588]
  Current humanoid teleoperation systems either lack reliable low-level control policies, or struggle to acquire accurate whole-body control commands.<n>We propose a novel humanoid teleoperation cockpit integrates a humanoid loco-manipulation policy and a low-cost exoskeleton-based hardware system.
  arXiv  Detail & Related papers  (2025-02-18T16:33:38Z)
• Human-Humanoid Robots Cross-Embodiment Behavior-Skill Transfer Using Decomposed Adversarial Learning from Demonstration [9.42179962375058]
  We propose a transferable framework that reduces the data bottleneck by using a unified digital human model as a common prototype.<n>The model learns behavior primitives from human demonstrations through adversarial imitation, and complex robot structures are decomposed into functional components.<n>Our framework is validated on five humanoid robots with diverse configurations.
  arXiv  Detail & Related papers  (2024-12-19T18:41:45Z)
• Grounding Language Models in Autonomous Loco-manipulation Tasks [3.8363685417355557]
  We propose a novel framework that learns, selects, and plans behaviors based on tasks in different scenarios. We leverage the planning and reasoning features of the large language model (LLM), constructing a hierarchical task graph. Experiments in simulation and real-world using the CENTAURO robot show that the language model based planner can efficiently adapt to new loco-manipulation tasks.
  arXiv  Detail & Related papers  (2024-09-02T15:27:48Z)
• HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation [7.01404330241523]
  HYPERmotion is a framework that learns, selects and plans behaviors based on tasks in different scenarios. We combine reinforcement learning with whole-body optimization to generate motion for 38 actuated joints. Experiments in simulation and real-world show that learned motions can efficiently adapt to new tasks.
  arXiv  Detail & Related papers  (2024-06-20T18:21:24Z)
• Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks [50.27313829438866]
  Plan-Seq-Learn (PSL) is a modular approach that uses motion planning to bridge the gap between abstract language and learned low-level control. PSL achieves success rates of over 85%, out-performing language-based, classical, and end-to-end approaches.
  arXiv  Detail & Related papers  (2024-05-02T17:59:31Z)
• Yell At Your Robot: Improving On-the-Fly from Language Corrections [84.09578841663195]
  We show that high-level policies can be readily supervised with human feedback in the form of language corrections. This framework enables robots not only to rapidly adapt to real-time language feedback, but also incorporate this feedback into an iterative training scheme.
  arXiv  Detail & Related papers  (2024-03-19T17:08:24Z)
• Never-Ending Behavior-Cloning Agent for Robotic Manipulation [38.756955029068294]
  NBAgent is a language-conditioned Never-ending Behavior-cloning Agent. It learns observation knowledge of novel 3D scene semantics and robot manipulation skills from skill-shared and skill-specific attributes.
  arXiv  Detail & Related papers  (2024-03-01T07:51:29Z)
• Skill Transformer: A Monolithic Policy for Mobile Manipulation [36.18813073796717]
  We present Skill Transformer, an approach for solving long-horizon robotic tasks by combining conditional sequence modeling and skill modularity. Conditioned on egocentric and proprioceptive observations of a robot, Skill Transformer is trained end-to-end to predict both a high-level skill and a whole-body low-level action. We test Skill Transformer on an embodied benchmark and find it performs robust task planning and low-level control in new scenarios, achieving a 2.5x higher success rate than baselines in hard rearrangement problems.
  arXiv  Detail & Related papers  (2023-08-19T01:37:41Z)
• Hierarchical generative modelling for autonomous robots [8.023920215148486]
  We show how a humanoid robot can autonomously complete a complex task that requires a holistic use of locomotion, manipulation, and grasping. Specifically, we demonstrate the ability of a humanoid robot that can retrieve and transport a box, open and walk through a door to reach the destination, approach and kick a football, while showing robust performance in presence of body damage and ground irregularities.
  arXiv  Detail & Related papers  (2023-08-15T13:51:03Z)
• Surfer: Progressive Reasoning with World Models for Robotic Manipulation [51.26109827779267]
  We introduce a novel and simple robot manipulation framework, called Surfer. Surfer treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene. It is based on the world model, treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene.
  arXiv  Detail & Related papers  (2023-06-20T07:06:04Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• Model Predictive Control for Fluid Human-to-Robot Handovers [50.72520769938633]
  Planning motions that take human comfort into account is not a part of the human-robot handover process. We propose to generate smooth motions via an efficient model-predictive control framework. We conduct human-to-robot handover experiments on a diverse set of objects with several users.
  arXiv  Detail & Related papers  (2022-03-31T23:08:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.