Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulation

• URL: http://arxiv.org/abs/2412.02676v2
• Date: Fri, 14 Feb 2025 22:01:30 GMT
• Title: Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulation
• Authors: Xuanlin Li, Tong Zhao, Xinghao Zhu, Jiuguang Wang, Tao Pang, Kuan Fang,
• Abstract summary: Generalizable Planning-Guided Diffusion Policy Learning (GLIDE) is an approach that learns to solve contact-rich bimanual manipulation tasks. We propose a set of essential design options in feature extraction, task representation, action prediction, and data augmentation. Our approach can enable a bimanual robotic system to effectively manipulate objects of diverse geometries, dimensions, and physical properties.
• Score: 16.244250979166214
• License:
• Abstract: Contact-rich bimanual manipulation involves precise coordination of two arms to change object states through strategically selected contacts and motions. Due to the inherent complexity of these tasks, acquiring sufficient demonstration data and training policies that generalize to unseen scenarios remain a largely unresolved challenge. Building on recent advances in planning through contacts, we introduce Generalizable Planning-Guided Diffusion Policy Learning (GLIDE), an approach that effectively learns to solve contact-rich bimanual manipulation tasks by leveraging model-based motion planners to generate demonstration data in high-fidelity physics simulation. Through efficient planning in randomized environments, our approach generates large-scale and high-quality synthetic motion trajectories for tasks involving diverse objects and transformations. We then train a task-conditioned diffusion policy via behavior cloning using these demonstrations. To tackle the sim-to-real gap, we propose a set of essential design options in feature extraction, task representation, action prediction, and data augmentation that enable learning robust prediction of smooth action sequences and generalization to unseen scenarios. Through experiments in both simulation and the real world, we demonstrate that our approach can enable a bimanual robotic system to effectively manipulate objects of diverse geometries, dimensions, and physical properties. Website: https://glide-manip.github.io/

Related papers

• Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos [61.925837909969815]
  We introduce Video2Policy, a novel framework that leverages internet RGB videos to reconstruct tasks based on everyday human behavior. Our method can successfully train RL policies on such tasks, including complex and challenging tasks such as throwing. We show that the generated simulation data can be scaled up for training a general policy, and it can be transferred back to the real robot in a Real2Sim2Real way.
  arXiv  Detail & Related papers  (2025-02-14T03:22:03Z)
• Semantic-Geometric-Physical-Driven Robot Manipulation Skill Transfer via Skill Library and Tactile Representation [6.324290412766366]
  skill library framework based on knowledge graphs endows robots with high-level skill awareness and spatial semantic understanding. At the motion level, an adaptive trajectory transfer method is developed using the A* algorithm and the skill library. At the physical level, we introduce an adaptive contour extraction and posture perception method based on tactile perception.
  arXiv  Detail & Related papers  (2024-11-18T16:42:07Z)
• Flex: End-to-End Text-Instructed Visual Navigation with Foundation Models [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies. Our findings are synthesized in Flex (Fly-lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors. We demonstrate the effectiveness of this approach on quadrotor fly-to-target tasks, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs [38.281562732050084]
  GenSim2 is a scalable framework for complex and realistic simulation task creation. The pipeline can generate data for up to 100 articulated tasks with 200 objects and reduce the required human efforts. We show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data.
  arXiv  Detail & Related papers  (2024-10-04T17:51:33Z)
• Learning Task Planning from Multi-Modal Demonstration for Multi-Stage Contact-Rich Manipulation [26.540648608911308]
  In this paper, we introduce an in-context learning framework that incorporates tactile and force-torque information from human demonstrations. We propose a bootstrapped reasoning pipeline that sequentially integrates each modality into a comprehensive task plan. This task plan is then used as a reference for planning in new task configurations.
  arXiv  Detail & Related papers  (2024-09-18T10:36:47Z)
• PoCo: Policy Composition from and for Heterogeneous Robot Learning [44.1315170137613]
  Current methods usually collect and pool all data from one domain to train a single policy. We present a flexible approach, dubbed Policy Composition, to combine information across diverse modalities and domains. Our method can use task-level composition for multi-task manipulation and be composed with analytic cost functions to adapt policy behaviors at inference time.
  arXiv  Detail & Related papers  (2024-02-04T14:51:49Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models. Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning. Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)
• Inferring Versatile Behavior from Demonstrations by Matching Geometric Descriptors [72.62423312645953]
  Humans intuitively solve tasks in versatile ways, varying their behavior in terms of trajectory-based planning and for individual steps. Current Imitation Learning algorithms often only consider unimodal expert demonstrations and act in a state-action-based setting. Instead, we combine a mixture of movement primitives with a distribution matching objective to learn versatile behaviors that match the expert's behavior and versatility.
  arXiv  Detail & Related papers  (2022-10-17T16:42:59Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.