Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation

• URL: http://arxiv.org/abs/2504.06961v1
• Date: Wed, 09 Apr 2025 15:12:38 GMT
• Title: Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation
• Abstract summary: Existing benchmarks and datasets predominantly focus on assembling geometric fragments or factory parts.<n>We present 2BY2, a large-scale annotated dataset for daily pairwise objects assembly.<n>We propose a two-step SE(3) pose estimation method with equivariant features for assembly constraints.
• Score: 29.02679318985968
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D assembly tasks, such as furniture assembly and component fitting, play a crucial role in daily life and represent essential capabilities for future home robots. Existing benchmarks and datasets predominantly focus on assembling geometric fragments or factory parts, which fall short in addressing the complexities of everyday object interactions and assemblies. To bridge this gap, we present 2BY2, a large-scale annotated dataset for daily pairwise objects assembly, covering 18 fine-grained tasks that reflect real-life scenarios, such as plugging into sockets, arranging flowers in vases, and inserting bread into toasters. 2BY2 dataset includes 1,034 instances and 517 pairwise objects with pose and symmetry annotations, requiring approaches that align geometric shapes while accounting for functional and spatial relationships between objects. Leveraging the 2BY2 dataset, we propose a two-step SE(3) pose estimation method with equivariant features for assembly constraints. Compared to previous shape assembly methods, our approach achieves state-of-the-art performance across all 18 tasks in the 2BY2 dataset. Additionally, robot experiments further validate the reliability and generalization ability of our method for complex 3D assembly tasks.

Related papers

• HeteroGenManip: Generalizable Manipulation For Heterogeneous Object Interactions [23.788389187142446]
  Generalizable manipulation involving cross-type object interactions is a critical yet challenging capability in robotics.<n>Existing foundation-model-based approaches often adopt end-to-end learning that obscures the distinction between these stages.<n>We propose HeteroGenManip, a task-conditioned, two-stage framework designed to decouple initial grasp from complex interaction execution.
  arXiv  Detail & Related papers  (2026-05-11T08:48:02Z)
• Articulated 3D Scene Graphs for Open-World Mobile Manipulation [55.97942733699124]
  We present MoMa-SG, a framework for building semantic-kinematic 3D scene graphs of articulated scenes.<n>We estimate articulation models using a novel unified twist estimation formulation.<n>We also introduce the novel Arti4D-Semantic dataset.
  arXiv  Detail & Related papers  (2026-02-18T10:40:35Z)
• Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations [112.29763628638112]
  Object-X is a versatile multi-modal 3D representation framework.<n>It can encoding rich object embeddings and decoding them back into geometric and visual reconstructions.<n>It supports a range of downstream tasks, including scene alignment, single-image 3D object reconstruction, and localization.
  arXiv  Detail & Related papers  (2025-06-05T09:14:42Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objects [67.96148051569993]
  This paper introduces a new neuro-implicit method that can reconstruct the geometry and appearance of two objects undergoing close interactions while disjoining both in 3D. The framework is end-to-end trainable and supervised using a novel alpha-blending regularisation. We introduce a new dataset consisting of close interactions between a human and an object and also evaluate on two scenes of humans performing martial arts.
  arXiv  Detail & Related papers  (2025-02-19T18:59:56Z)
• Holistic Understanding of 3D Scenes as Universal Scene Description [56.69740649781989]
  3D scene understanding is a long-standing challenge in computer vision and a key component in enabling mixed reality, wearable computing, and embodied AI. We introduce an expertly curated dataset in the Universal Scene Description (USD) format featuring high-quality manual annotations. With its broad and high-quality annotations, the data provides the basis for holistic 3D scene understanding models.
  arXiv  Detail & Related papers  (2024-12-02T11:33:55Z)
• GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding [53.42728468191711]
  Open-Vocabulary 3D object affordance grounding aims to anticipate action possibilities'' regions on 3D objects with arbitrary instructions. We propose GREAT (GeometRy-intEntion collAboraTive inference) for Open-Vocabulary 3D Object Affordance Grounding.
  arXiv  Detail & Related papers  (2024-11-29T11:23:15Z)
• OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion [39.14950571922401]
  OAKINK2 is a dataset of bimanual object manipulation tasks for complex daily activities. It introduces three level of abstraction to organize the manipulation tasks: Affordance, Primitive Task, and Complex Task. OakINK2 dataset provides multi-view image streams and precise pose annotations for the human body, hands and various interacting objects.
  arXiv  Detail & Related papers  (2024-03-28T13:47:19Z)
• SM$^3$: Self-Supervised Multi-task Modeling with Multi-view 2D Images for Articulated Objects [24.737865259695006]
  We propose a self-supervised interaction perception method, referred to as SM$3$, to model articulated objects. By constructing 3D geometries and textures from the captured 2D images, SM$3$ achieves integrated optimization of movable part and joint parameters. Evaluations demonstrate that SM$3$ surpasses existing benchmarks across various categories and objects, while its adaptability in real-world scenarios has been thoroughly validated.
  arXiv  Detail & Related papers  (2024-01-17T11:15:09Z)
• Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers [65.51132104404051]
  We introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
  arXiv  Detail & Related papers  (2023-12-13T14:27:45Z)
• Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion [110.84357383258818]
  We propose a novel approach to lift 2D segments to 3D and fuse them by means of a neural field representation. The core of our approach is a slow-fast clustering objective function, which is scalable and well-suited for scenes with a large number of objects. Our approach outperforms the state-of-the-art on challenging scenes from the ScanNet, Hypersim, and Replica datasets.
  arXiv  Detail & Related papers  (2023-06-07T17:57:45Z)
• Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Clouds [28.330364666426345]
  We build rearticulable models for arbitrary everyday man-made objects containing an arbitrary number of parts. Our method identifies the distinct object parts, what parts are connected to what other parts, and the properties of the joints connecting each part pair.
  arXiv  Detail & Related papers  (2023-06-01T17:59:21Z)
• Lifelong Ensemble Learning based on Multiple Representations for Few-Shot Object Recognition [6.282068591820947]
  We present a lifelong ensemble learning approach based on multiple representations to address the few-shot object recognition problem. To facilitate lifelong learning, each approach is equipped with a memory unit for storing and retrieving object information instantly. We have performed extensive sets of experiments to assess the performance of the proposed approach in offline, and open-ended scenarios.
  arXiv  Detail & Related papers  (2022-05-04T10:29:10Z)
• Towards Robotic Assembly by Predicting Robust, Precise and Task-oriented Grasps [17.07993278175686]
  We propose a method to optimize for grasp, precision, and task performance by learning three cascaded networks. We evaluate our method in simulation on three common assembly tasks: inserting gears onto pegs, aligning brackets into corners, and inserting shapes into slots.
  arXiv  Detail & Related papers  (2020-11-04T18:29:01Z)
• The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose [108.21037046507483]
  IKEA ASM is a three million frame, multi-view, furniture assembly video dataset that includes depth, atomic actions, object segmentation, and human pose. We benchmark prominent methods for video action recognition, object segmentation and human pose estimation tasks on this challenging dataset. The dataset enables the development of holistic methods, which integrate multi-modal and multi-view data to better perform on these tasks.
  arXiv  Detail & Related papers  (2020-07-01T11:34:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.