AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulation

• URL: http://arxiv.org/abs/2412.06779v2
• Date: Thu, 27 Mar 2025 02:34:48 GMT
• Title: AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulation
• Abstract summary: We propose a plug-and-play method named AnyBimanual, which transfers pre-trained unimanual policy to general bimanual manipulation policy.<n>We show that AnyBimanual shows superiority on 12 simulated tasks with a sizable 12.67% improvement in success rate over previous methods.
• Score: 15.44403761987512
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Performing general language-conditioned bimanual manipulation tasks is of great importance for many applications ranging from household service to industrial assembly. However, collecting bimanual manipulation data is expensive due to the high-dimensional action space, which poses challenges for conventional methods to handle general bimanual manipulation tasks. In contrast, unimanual policy has recently demonstrated impressive generalizability across a wide range of tasks because of scaled model parameters and training data, which can provide sharable manipulation knowledge for bimanual systems. To this end, we propose a plug-and-play method named AnyBimanual, which transfers pre-trained unimanual policy to general bimanual manipulation policy with few bimanual demonstrations. Specifically, we first introduce a skill manager to dynamically schedule the skill representations discovered from pre-trained unimanual policy for bimanual manipulation tasks, which linearly combines skill primitives with task-oriented compensation to represent the bimanual manipulation instruction. To mitigate the observation discrepancy between unimanual and bimanual systems, we present a visual aligner to generate soft masks for visual embedding of the workspace, which aims to align visual input of unimanual policy model for each arm with those during pretraining stage. AnyBimanual shows superiority on 12 simulated tasks from RLBench2 with a sizable 12.67% improvement in success rate over previous methods. Experiments on 9 real-world tasks further verify its practicality with an average success rate of 84.62%.

Related papers

• Composing Dextrous Grasping and In-hand Manipulation via Scoring with a Reinforcement Learning Critic [7.759447374181355]
  In-hand manipulation and grasping are fundamental yet often separately addressed tasks in robotics.<n>We propose a method for bridging this gap by leveraging the critic network of a reinforcement learning agent trained for in-hand manipulation.<n>Our experiments show that this method significantly increases the success rate of in-hand manipulation without requiring additional training.
  arXiv  Detail & Related papers  (2025-05-19T15:36:34Z)
• Imitation Learning with Precisely Labeled Human Demonstrations [0.0]
  This work builds on prior studies that demonstrate the viability of using hand-held grippers for efficient data collection. We leverage the user's control over the gripper's appearance--specifically by assigning it a unique, easily segmentable color--to enable precise end-effector pose estimation. We show in simulation that precisely labeled human demonstrations on their own allow policies to reach on average 88.1% of the performance of using robot demonstrations.
  arXiv  Detail & Related papers  (2025-04-18T17:12:00Z)
• VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation [8.882764358932276]
  Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination. Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills. We introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation.
  arXiv  Detail & Related papers  (2025-01-07T08:14:53Z)
• PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks [32.94163186662167]
  Bimanual manipulation is challenging due to precise spatial and temporal coordination required between two arms. This paper addresses the gap by extending RLBench to bimanual manipulation. We open-source our code and benchmark comprising 13 new tasks with 23 unique task variations.
  arXiv  Detail & Related papers  (2024-06-29T02:06:01Z)
• NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation [21.02437461550044]
  Many real-world tasks demand intricate multi-step reasoning. We introduce a benchmark, NrVLM, comprising 15 distinct manipulation tasks. We propose a novel learning framework that completes the manipulation task step-by-step according to the fine-grained instructions.
  arXiv  Detail & Related papers  (2024-03-13T09:12:16Z)
• Multi-task real-robot data with gaze attention for dual-arm fine manipulation [4.717749411286867]
  This paper introduces a dataset of diverse object manipulations that includes dual-arm tasks and/or tasks requiring fine manipulation. We have generated dataset with 224k episodes (150 hours, 1,104 language instructions) which includes dual-arm fine tasks such as bowl-moving, pencil-case opening or banana-peeling. This dataset includes visual attention signals as well as dual-action labels, a signal that separates actions into a robust reaching trajectory and precise interaction with objects, and language instructions to achieve robust and precise object manipulation.
  arXiv  Detail & Related papers  (2024-01-15T11:20:34Z)
• Self-supervised Optimization of Hand Pose Estimation using Anatomical Features and Iterative Learning [4.698846136465861]
  This paper presents a self-supervised pipeline for adapting hand pose estimation to specific use cases with minimal human interaction. The pipeline consists of a general machine learning model for hand pose estimation trained on a generalized dataset. The effectiveness of the pipeline is demonstrated by training an activity recognition as a downstream task in the manual assembly scenario.
  arXiv  Detail & Related papers  (2023-07-06T14:13:11Z)
• Instruction-driven history-aware policies for robotic manipulations [82.25511767738224]
  We propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations. We evaluate our method on the challenging RLBench benchmark and on a real-world robot.
  arXiv  Detail & Related papers  (2022-09-11T16:28:25Z)
• Learning Action Conditions from Instructional Manuals for Instruction Understanding [48.52663250368341]
  We propose a task dubbed action condition inference, and collecting a high-quality, human annotated dataset of preconditions and postconditions of actions in instructional manuals. We propose a weakly supervised approach to automatically construct large-scale training instances from online instructional manuals, and curate a densely human-annotated and validated dataset to study how well the current NLP models can infer action-condition dependencies in instruction texts.
  arXiv  Detail & Related papers  (2022-05-25T00:19:59Z)
• ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters [123.88692739360457]
  General-purpose motor skills enable humans to perform complex tasks. These skills also provide powerful priors for guiding their behaviors when learning new tasks. We present a framework for learning versatile and reusable skill embeddings for physically simulated characters.
  arXiv  Detail & Related papers  (2022-05-04T06:13:28Z)
• Transfer Learning in Conversational Analysis through Reusing Preprocessing Data as Supervisors [52.37504333689262]
  Using noisy labels in single-task learning increases the risk of over-fitting. Auxiliary tasks could improve the performance of the primary task learning during the same training.
  arXiv  Detail & Related papers  (2021-12-02T08:40:42Z)
• Playful Interactions for Representation Learning [82.59215739257104]
  We propose to use playful interactions in a self-supervised manner to learn visual representations for downstream tasks. We collect 2 hours of playful data in 19 diverse environments and use self-predictive learning to extract visual representations. Our representations generalize better than standard behavior cloning and can achieve similar performance with only half the number of required demonstrations.
  arXiv  Detail & Related papers  (2021-07-19T17:54:48Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.