InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation

• URL: http://arxiv.org/abs/2409.07914v3
• Date: Wed, 16 Oct 2024 08:52:42 GMT
• Title: InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation
• Abstract summary: We introduce InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers. InterACT is a novel imitation learning framework designed specifically for bimanual manipulation. Our experiments, conducted on various simulated and real-world bimanual manipulation tasks, demonstrate that InterACT outperforms existing methods.
• Score: 1.4835219023165949
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bimanual manipulation presents unique challenges compared to unimanual tasks due to the complexity of coordinating two robotic arms. In this paper, we introduce InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation. InterACT leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs. The framework comprises a Hierarchical Attention Encoder, which processes multi-modal inputs through segment-wise and cross-segment attention mechanisms, and a Multi-arm Decoder that generates each arm's action predictions in parallel, while sharing information between the arms through synchronization blocks by providing the other arm's intermediate output as context. Our experiments, conducted on various simulated and real-world bimanual manipulation tasks, demonstrate that InterACT outperforms existing methods. Detailed ablation studies further validate the significance of key components, including the impact of CLS tokens, cross-segment encoders, and synchronization blocks on task performance. We provide supplementary materials and videos on our project page.

Related papers

• See Selectively, Act Adaptively: Dual-Level Structural Decomposition for Bimanual Robot Manipulation [1.7587442088965224]
  We propose a bimanual manipulation framework based on Dual-Level Structural Decomposition.<n>We evaluate the proposed method on six simulated bimanual manipulation tasks in RoboTwin 2.0 and three long-horizon real-world tasks.
  arXiv  Detail & Related papers  (2026-06-11T12:33:55Z)
• Information-Theoretic Detection of Bimanual Interactions for Dual-Arm Robot Plan Generation [14.053079591358797]
  This paper presents a novel one-shot method for processing a single RGB video of a bimanual task demonstration to generate an execution plan for a dual-arm robotic system.<n>We apply Shannon's information theory to analyze the information flow between scene elements and leverage scene graph properties.<n>The generated plan is a modular behavior tree that assumes different structures based on the desired arms coordination.
  arXiv  Detail & Related papers  (2026-01-27T17:38:58Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesis [20.922897975281316]
  We propose a dual-stream neural framework to generate synchronized hand gestures for piano playing from audio input. A Hand-Coordinated Asymmetric Attention mechanism suppresses symmetric (common-mode) noise to highlight asymmetric hand-specific features.
  arXiv  Detail & Related papers  (2025-04-14T05:17:41Z)
• Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers [4.119006369973485]
  We focus on coordination and efficiency between both arms, particularly synchronized actions. We propose a novel imitation learning architecture that predicts cooperative actions. Our model demonstrated a high success rate for comparison and suggested a suitable architecture for the policy learning of bimanual manipulation.
  arXiv  Detail & Related papers  (2025-03-18T05:20:34Z)
• Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Framework [51.39847596489193]
  Bimanual robotic manipulation is an emerging and critical topic in the robotics community.<n>We propose a novel decoupled interaction framework that considers the characteristics of different tasks in bimanual manipulation.<n>Our framework achieves outstanding performance, with a 23.5% boost over the SOTA method.
  arXiv  Detail & Related papers  (2025-03-12T09:28:41Z)
• DeepInteraction++: Multi-Modality Interaction for Autonomous Driving [80.8837864849534]
  We introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout. DeepInteraction++ is a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:04:21Z)
• EAGER: Two-Stream Generative Recommender with Behavior-Semantic Collaboration [63.112790050749695]
  We introduce EAGER, a novel generative recommendation framework that seamlessly integrates both behavioral and semantic information. We validate the effectiveness of EAGER on four public benchmarks, demonstrating its superior performance compared to existing methods.
  arXiv  Detail & Related papers  (2024-06-20T06:21:56Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• Automatic Interaction and Activity Recognition from Videos of Human Manual Demonstrations with Application to Anomaly Detection [0.0]
  This paper exploits Scene Graphs to extract key interaction features from image sequences while simultaneously motion patterns and context. The method introduces event-based automatic video segmentation and clustering, which allow for the grouping of similar events and detect if a monitored activity is executed correctly.
  arXiv  Detail & Related papers  (2023-04-19T16:15:23Z)
• Object Discovery from Motion-Guided Tokens [50.988525184497334]
  We augment the auto-encoder representation learning framework with motion-guidance and mid-level feature tokenization. Our approach enables the emergence of interpretable object-specific mid-level features.
  arXiv  Detail & Related papers  (2023-03-27T19:14:00Z)
• CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spaces [9.578169216444813]
  This paper proposes an approach to coordinating multi-robot manipulation through learned latent action spaces that are shared across different agents. We validate our method in simulated multi-robot manipulation tasks and demonstrate improvement over previous baselines in terms of sample efficiency and learning performance.
  arXiv  Detail & Related papers  (2022-11-28T23:20:47Z)
• Context-Aware Interaction Network for Question Matching [51.76812857301819]
  We propose a context-aware interaction network (COIN) to align two sequences and infer their semantic relationship. Specifically, each interaction block includes (1) a context-aware cross-attention mechanism to effectively integrate contextual information, and (2) a gate fusion layer to flexibly interpolate aligned representations.
  arXiv  Detail & Related papers  (2021-04-17T05:03:56Z)
• Pedestrian Behavior Prediction via Multitask Learning and Categorical Interaction Modeling [13.936894582450734]
  We propose a multitask learning framework that simultaneously predicts trajectories and actions of pedestrians by relying on multimodal data. We show that our model achieves state-of-the-art performance and improves trajectory and action prediction by up to 22% and 6% respectively.
  arXiv  Detail & Related papers  (2020-12-06T15:57:11Z)
• Cascaded Human-Object Interaction Recognition [175.60439054047043]
  We introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding.
  arXiv  Detail & Related papers  (2020-03-09T17:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.