QuadWBG: Generalizable Quadrupedal Whole-Body Grasping

• URL: http://arxiv.org/abs/2411.06782v1
• Date: Mon, 11 Nov 2024 08:19:54 GMT
• Title: QuadWBG: Generalizable Quadrupedal Whole-Body Grasping
• Authors: Jilong Wang, Javokhirbek Rajabov, Chaoyi Xu, Yiming Zheng, He Wang,
• Abstract summary: We present a modular framework for robust whole-body loco-manipulation controller based on a single arm-mounted camera. The proposed system achieves state-of-the-art one-time grasping accuracy of 89% in the real world.
• Score: 7.802964645500815
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Legged robots with advanced manipulation capabilities have the potential to significantly improve household duties and urban maintenance. Despite considerable progress in developing robust locomotion and precise manipulation methods, seamlessly integrating these into cohesive whole-body control for real-world applications remains challenging. In this paper, we present a modular framework for robust and generalizable whole-body loco-manipulation controller based on a single arm-mounted camera. By using reinforcement learning (RL), we enable a robust low-level policy for command execution over 5 dimensions (5D) and a grasp-aware high-level policy guided by a novel metric, Generalized Oriented Reachability Map (GORM). The proposed system achieves state-of-the-art one-time grasping accuracy of 89% in the real world, including challenging tasks such as grasping transparent objects. Through extensive simulations and real-world experiments, we demonstrate that our system can effectively manage a large workspace, from floor level to above body height, and perform diverse whole-body loco-manipulation tasks.

Related papers

• ModSkill: Physical Character Skill Modularization [21.33764810227885]
  We introduce a novel skill learning framework, ModSkill, that decouples complex full-body skills into compositional, modular skills for independent body parts. Our results show that this modularized skill learning framework, enhanced by generative sampling, outperforms existing methods in precise full-body motion tracking.
  arXiv  Detail & Related papers  (2025-02-19T22:55:49Z)
• Learning Multi-Agent Loco-Manipulation for Long-Horizon Quadrupedal Pushing [33.689150109924526]
  This paper tackles the task of obstacle-aware, long-horizon pushing by multiple quadrupedal robots. We propose a hierarchical multi-agent reinforcement learning framework with three levels of control.
  arXiv  Detail & Related papers  (2024-11-11T16:27:25Z)
• Whole-body end-effector pose tracking [10.426087117345096]
  We introduce a whole-body RL formulation for end-effector pose tracking in a large workspace on rough, unstructured terrains. Our proposed method involves a terrain-aware sampling strategy for the robot's initial configuration and end-effector pose commands. On deployment, it achieves a pose-tracking error of 2.64 cm and 3.64 degrees, outperforming existing competitive baselines.
  arXiv  Detail & Related papers  (2024-09-24T12:51:32Z)
• QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds [51.05639500325598]
  QuadrupedGPT is a versatile agent designed to master a broad range of complex tasks with agility comparable to that of a pet. Our agent processes human command and environmental contexts using a large multimodal model (LMM) It is equipped with problem-solving capabilities that enable it to decompose long-term goals into a sequence of executable subgoals.
  arXiv  Detail & Related papers  (2024-06-24T12:14:24Z)
• Modular Neural Network Policies for Learning In-Flight Object Catching with a Robot Hand-Arm System [55.94648383147838]
  We present a modular framework designed to enable a robot hand-arm system to learn how to catch flying objects. Our framework consists of five core modules: (i) an object state estimator that learns object trajectory prediction, (ii) a catching pose quality network that learns to score and rank object poses for catching, (iii) a reaching control policy trained to move the robot hand to pre-catch poses, and (iv) a grasping control policy trained to perform soft catching motions. We conduct extensive evaluations of our framework in simulation for each module and the integrated system, to demonstrate high success rates of in-flight
  arXiv  Detail & Related papers  (2023-12-21T16:20:12Z)
• RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking [54.776890150458385]
  We develop an efficient system for training universal agents capable of multi-task manipulation skills. We are able to train a single agent capable of 12 unique skills, and demonstrate its generalization over 38 tasks. On average, RoboAgent outperforms prior methods by over 40% in unseen situations.
  arXiv  Detail & Related papers  (2023-09-05T03:14:39Z)
• Learning and Adapting Agile Locomotion Skills by Transferring Experience [71.8926510772552]
  We propose a framework for training complex robotic skills by transferring experience from existing controllers to jumpstart learning new tasks. We show that our method enables learning complex agile jumping behaviors, navigating to goal locations while walking on hind legs, and adapting to new environments.
  arXiv  Detail & Related papers  (2023-04-19T17:37:54Z)
• Bi-Manual Block Assembly via Sim-to-Real Reinforcement Learning [24.223788665601678]
  Two xArm6 robots solve the U-shape assembly task with a success rate of above90% in simulation, and 50% on real hardware without any additional real-world fine-tuning. Our results present a significant step forward for bi-arm capability on real hardware, and we hope our system can inspire future research on deep RL and Sim2Real transfer bi-manualpolicies.
  arXiv  Detail & Related papers  (2023-03-27T01:25:24Z)
• Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance [71.36749876465618]
  We describe a system for vision-based dexterous manipulation that provides a "programming-free" approach for users to define new tasks. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples. experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world.
  arXiv  Detail & Related papers  (2022-12-19T22:50:40Z)
• Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks [26.445959214209505]
  This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks. The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant workspace.
  arXiv  Detail & Related papers  (2021-06-26T17:54:16Z)
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation [99.2543521972137]
  ReLMoGen is a framework that combines a learned policy to predict subgoals and a motion generator to plan and execute the motion needed to reach these subgoals. Our method is benchmarked on a diverse set of seven robotics tasks in photo-realistic simulation environments. ReLMoGen shows outstanding transferability between different motion generators at test time, indicating a great potential to transfer to real robots.
  arXiv  Detail & Related papers  (2020-08-18T08:05:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.