PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks

• URL: http://arxiv.org/abs/2407.00278v2
• Date: Wed, 31 Jul 2024 17:57:37 GMT
• Title: PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks
• Abstract summary: Bimanual manipulation is challenging due to precise spatial and temporal coordination required between two arms. This paper addresses the gap by extending RLBench to bimanual manipulation. We open-source our code and benchmark comprising 13 new tasks with 23 unique task variations.
• Score: 32.94163186662167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bimanual manipulation is challenging due to precise spatial and temporal coordination required between two arms. While there exist several real-world bimanual systems, there is a lack of simulated benchmarks with a large task diversity for systematically studying bimanual capabilities across a wide range of tabletop tasks. This paper addresses the gap by extending RLBench to bimanual manipulation. We open-source our code and benchmark comprising 13 new tasks with 23 unique task variations, each requiring a high degree of coordination and adaptability. To kickstart the benchmark, we extended several state-of-the art methods to bimanual manipulation and also present a language-conditioned behavioral cloning agent -- PerAct2, which enables the learning and execution of bimanual 6-DoF manipulation tasks. Our novel network architecture efficiently integrates language processing with action prediction, allowing robots to understand and perform complex bimanual tasks in response to user-specified goals. Project website with code is available at: http://bimanual.github.io

Related papers

• DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo [82.73355932907565]
  DexJoCo is a benchmark and toolkit for task-oriented dexterous manipulation.<n>We present 11 functionally grounded tasks that evaluate tool-use, bimanual coordination, long-horizon execution, and reasoning.<n>We benchmark modern models under diverse settings, including visual and dynamics randomization, multi-task training, and action-head adaptation.
  arXiv  Detail & Related papers  (2026-05-15T17:59:51Z)
• BiCoord: A Bimanual Manipulation Benchmark towards Long-Horizon Spatial-Temporal Coordination [20.953323366922266]
  BiCoord is a benchmark for long-horizon and tightly coordinated bimanual manipulation.<n>Specifically, BiCoord comprises diverse tasks that require continuous inter-arm dependency and dynamic role exchange.<n>Results show that representative manipulation policies struggle with long-duration and highly coupled tasks.
  arXiv  Detail & Related papers  (2026-04-07T13:02:17Z)
• MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Language-Conditioned Mixture-of-Experts Transformers [3.890941830250993]
  We propose a lightweight multi-task imitation learning framework for bimanual manipulation.<n>MoE-ACT integrates sparse Mixture-of-Experts (MoE) modules into the Transformer encoder of ACT.<n>We show that MoE-ACT substantially improves multi-task performance.
  arXiv  Detail & Related papers  (2026-03-16T13:33:59Z)
• Instruct2Act: From Human Instruction to Actions Sequencing and Execution via Robot Action Network for Robotic Manipulation [14.833622989644352]
  We develop a lightweight, fully on-device pipeline that converts natural-language commands into reliable manipulation.<n>Instruct2Act attains 91.5% sub-actions prediction accuracy while retaining a small footprint.<n>Results demonstrate that fine-grained instruction-to-action parsing, coupled with DATRN-based trajectory generation and vision-guided grounding, provides a practical path to deterministic, real-time manipulation.
  arXiv  Detail & Related papers  (2026-02-10T16:25:39Z)
• BiKC+: Bimanual Hierarchical Imitation with Keypose-Conditioned Coordination-Aware Consistency Policies [25.25845809170421]
  We propose a novel keypose-conditioned coordination-aware consistency policy tailored for bimanual manipulation.<n>Our framework instantiates hierarchical imitation learning with a high-level keypose predictor and a low-level trajectory generator.<n>In particular, we devise an innovative approach for identifying bimanual keyposes, considering both robot-centric action features and task-centric operation styles.
  arXiv  Detail & Related papers  (2026-01-17T17:33:18Z)
• Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Framework [51.39847596489193]
  Bimanual robotic manipulation is an emerging and critical topic in the robotics community.<n>We propose a novel decoupled interaction framework that considers the characteristics of different tasks in bimanual manipulation.<n>Our framework achieves outstanding performance, with a 23.5% boost over the SOTA method.
  arXiv  Detail & Related papers  (2025-03-12T09:28:41Z)
• AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulation [15.44403761987512]
  We propose a plug-and-play method named AnyBimanual, which transfers pre-trained unimanual policy to general bimanual manipulation policy. We show that AnyBimanual shows superiority on 12 simulated tasks with a sizable 12.67% improvement in success rate over previous methods.
  arXiv  Detail & Related papers  (2024-12-09T18:58:43Z)
• Reinforcement Learning with Action Sequence for Data-Efficient Robot Learning [62.3886343725955]
  We introduce a novel RL algorithm that learns a critic network that outputs Q-values over a sequence of actions. By explicitly training the value functions to learn the consequence of executing a series of current and future actions, our algorithm allows for learning useful value functions from noisy trajectories.
  arXiv  Detail & Related papers  (2024-11-19T01:23:52Z)
• BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulation [48.08416841005715]
  We introduce a novel keypose-conditioned consistency policy tailored for bimanual manipulation. It is a hierarchical imitation learning framework that consists of a high-level keypose predictor and a low-level trajectory generator. Simulated and real-world experimental results demonstrate that the proposed approach surpasses baseline methods in terms of success rate and operational efficiency.
  arXiv  Detail & Related papers  (2024-06-14T14:49:12Z)
• Large Language Models for Orchestrating Bimanual Robots [19.60907949776435]
  We present LAnguage-model-based Bimanual ORchestration (LABOR) to analyze task configurations and devise coordination control policies. We evaluate our method through simulated experiments involving two classes of long-horizon tasks using the NICOL humanoid robot.
  arXiv  Detail & Related papers  (2024-04-02T15:08:35Z)
• LEMMA: Learning Language-Conditioned Multi-Robot Manipulation [21.75163634731677]
  LanguagE-Conditioned Multi-robot MAnipulation (LEMMA) LeMMA features 8 types of procedurally generated tasks with varying degree of complexity. For each task, we provide 800 expert demonstrations and human instructions for training and evaluations.
  arXiv  Detail & Related papers  (2023-08-02T04:37:07Z)
• RObotic MAnipulation Network (ROMAN) $\unicode{x2013}$ Hybrid Hierarchical Learning for Solving Complex Sequential Tasks [70.69063219750952]
  We present a Hybrid Hierarchical Learning framework, the Robotic Manipulation Network (ROMAN) ROMAN achieves task versatility and robust failure recovery by integrating behavioural cloning, imitation learning, and reinforcement learning. Experimental results show that by orchestrating and activating these specialised manipulation experts, ROMAN generates correct sequential activations for accomplishing long sequences of sophisticated manipulation tasks.
  arXiv  Detail & Related papers  (2023-06-30T20:35:22Z)
• Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model [63.66204449776262]
  Instruct2Act is a framework that maps multi-modal instructions to sequential actions for robotic manipulation tasks. Our approach is adjustable and flexible in accommodating various instruction modalities and input types. Our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks.
  arXiv  Detail & Related papers  (2023-05-18T17:59:49Z)
• Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks [21.65346551790888]
  DeL-TaCo is a method for conditioning a robotic policy on task embeddings comprised of two components: a visual demonstration and a language instruction. To our knowledge, this is the first work to show that simultaneously conditioning a multi-task robotic manipulation policy on both demonstration and language embeddings improves sample efficiency and generalization over conditioning on either modality alone.
  arXiv  Detail & Related papers  (2022-10-10T08:06:58Z)
• VIMA: General Robot Manipulation with Multimodal Prompts [82.01214865117637]
  We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts. We develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively.
  arXiv  Detail & Related papers  (2022-10-06T17:50:11Z)
• Instruction-driven history-aware policies for robotic manipulations [82.25511767738224]
  We propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations. We evaluate our method on the challenging RLBench benchmark and on a real-world robot.
  arXiv  Detail & Related papers  (2022-09-11T16:28:25Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Disentangled Attention as Intrinsic Regularization for Bimanual Multi-Object Manipulation [18.38312133753365]
  We address the problem of solving complex bimanual robot manipulation tasks on multiple objects with sparse rewards. We propose a novel technique called disentangled attention, which provides an intrinsic regularization for two robots to focus on separate sub-tasks and objects. Experimental results show that our proposed intrinsic regularization successfully avoids domination and reduces conflicts for the policies.
  arXiv  Detail & Related papers  (2021-06-10T16:53:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.