GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts

• URL: http://arxiv.org/abs/2211.05272v2
• Date: Sun, 26 Mar 2023 23:59:07 GMT
• Title: GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
• Authors: Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, He Wang
• Abstract summary: We learn cross-category skills via Generalizable and Actionable Parts (GAParts) Based on GAPartNet, we investigate three cross-category tasks: part segmentation, part pose estimation, and part-based object manipulation. Our method outperforms all existing methods by a large margin, no matter on seen or unseen categories.
• Score: 28.922958261132475
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For years, researchers have been devoted to generalizable object perception and manipulation, where cross-category generalizability is highly desired yet underexplored. In this work, we propose to learn such cross-category skills via Generalizable and Actionable Parts (GAParts). By identifying and defining 9 GAPart classes (lids, handles, etc.) in 27 object categories, we construct a large-scale part-centric interactive dataset, GAPartNet, where we provide rich, part-level annotations (semantics, poses) for 8,489 part instances on 1,166 objects. Based on GAPartNet, we investigate three cross-category tasks: part segmentation, part pose estimation, and part-based object manipulation. Given the significant domain gaps between seen and unseen object categories, we propose a robust 3D segmentation method from the perspective of domain generalization by integrating adversarial learning techniques. Our method outperforms all existing methods by a large margin, no matter on seen or unseen categories. Furthermore, with part segmentation and pose estimation results, we leverage the GAPart pose definition to design part-based manipulation heuristics that can generalize well to unseen object categories in both the simulator and the real world. Our dataset, code, and demos are available on our project page.

Related papers

• PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects [104.34288029037141]
  We present PartGLEE, a part-level foundation model for locating and identifying both objects and parts in images. PartGLEE accomplishes detection, segmentation, and grounding of instances at any granularity in the open world scenario.
  arXiv  Detail & Related papers  (2024-07-23T17:58:26Z)
• OMG-Seg: Is One Model Good Enough For All Segmentation? [83.17068644513144]
  OMG-Seg is a transformer-based encoder-decoder architecture with task-specific queries and outputs. We show that OMG-Seg can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead.
  arXiv  Detail & Related papers  (2024-01-18T18:59:34Z)
• Towards Open-World Segmentation of Parts [16.056921233445784]
  We propose to explore a class-agnostic part segmentation task. We argue that models trained without part classes can better localize parts and segment them on objects unseen in training. We show notable and consistent gains by our approach, essentially a critical step towards open-world part segmentation.
  arXiv  Detail & Related papers  (2023-05-26T10:34:58Z)
• Semi-Weakly Supervised Object Kinematic Motion Prediction [56.282759127180306]
  Given a 3D object, kinematic motion prediction aims to identify the mobile parts as well as the corresponding motion parameters. We propose a graph neural network to learn the map between hierarchical part-level segmentation and mobile parts parameters. The network predictions yield a large scale of 3D objects with pseudo labeled mobility information.
  arXiv  Detail & Related papers  (2023-03-31T02:37:36Z)
• PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations [12.552149411655355]
  We build the first large-scale, part-based cross-category object manipulation benchmark, PartManip. We train a state-based expert with our proposed part-based canonicalization and part-aware rewards, and then distill the knowledge to a vision-based student. For cross-category generalization, we introduce domain adversarial learning for domain-invariant feature extraction.
  arXiv  Detail & Related papers  (2023-03-29T18:29:30Z)
• Towards Robust Part-aware Instance Segmentation for Industrial Bin Picking [113.79582950811348]
  We formulate a novel part-aware instance segmentation pipeline for industrial bin picking. We design a part-aware network to predict part masks and part-to-part offsets, followed by a part aggregation module to assemble the recognized parts into instances. We contribute the first instance segmentation dataset, which contains a variety of industrial objects that are thin and have non-trivial shapes.
  arXiv  Detail & Related papers  (2022-03-05T14:58:05Z)
• 3D Compositional Zero-shot Learning with DeCompositional Consensus [102.7571947144639]
  We argue that part knowledge should be composable beyond the observed object classes. We present 3D Compositional Zero-shot Learning as a problem of part generalization from seen to unseen object classes.
  arXiv  Detail & Related papers  (2021-11-29T16:34:53Z)
• CGPart: A Part Segmentation Dataset Based on 3D Computer Graphics Models [19.691187561807475]
  CGPart provides detailed annotations on 3D CAD models, synthetic images, and real test images. CGPart includes $21$ 3D CAD models covering $5$ vehicle categories, each with detailed per-mesh part labeling. We make $168,000$ synthetic images from these CAD models, each with automatically generated part segmentation ground-truth.
  arXiv  Detail & Related papers  (2021-03-25T19:34:21Z)
• Commonality-Parsing Network across Shape and Appearance for Partially Supervised Instance Segmentation [71.59275788106622]
  We propose to learn the underlying class-agnostic commonalities that can be generalized from mask-annotated categories to novel categories. Our model significantly outperforms the state-of-the-art methods on both partially supervised setting and few-shot setting for instance segmentation on COCO dataset.
  arXiv  Detail & Related papers  (2020-07-24T07:23:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.