DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Demo

• URL: http://arxiv.org/abs/2412.05268v1
• Date: Fri, 06 Dec 2024 18:55:09 GMT
• Title: DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Demo
• Authors: Junzhe Zhu, Yuanchen Ju, Junyi Zhang, Muhan Wang, Zhecheng Yuan, Kaizhe Hu, Huazhe Xu,
• Abstract summary: We present DenseMatcher, a method capable of computing 3D correspondences between in-the-wild objects that share similar structures. DenseMatcher significantly outperforms prior 3D matching baselines by 43.5%.
• Score: 22.659984212937907
• License:
• Abstract: Dense 3D correspondence can enhance robotic manipulation by enabling the generalization of spatial, functional, and dynamic information from one object to an unseen counterpart. Compared to shape correspondence, semantic correspondence is more effective in generalizing across different object categories. To this end, we present DenseMatcher, a method capable of computing 3D correspondences between in-the-wild objects that share similar structures. DenseMatcher first computes vertex features by projecting multiview 2D features onto meshes and refining them with a 3D network, and subsequently finds dense correspondences with the obtained features using functional map. In addition, we craft the first 3D matching dataset that contains colored object meshes across diverse categories. In our experiments, we show that DenseMatcher significantly outperforms prior 3D matching baselines by 43.5%. We demonstrate the downstream effectiveness of DenseMatcher in (i) robotic manipulation, where it achieves cross-instance and cross-category generalization on long-horizon complex manipulation tasks from observing only one demo; (ii) zero-shot color mapping between digital assets, where appearance can be transferred between different objects with relatable geometry.

Related papers

• View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields [52.08335264414515]
  We learn a novel feature field within a Neural Radiance Field (NeRF) representing a 3D scene. Our method takes view-inconsistent multi-granularity 2D segmentations as input and produces a hierarchy of 3D-consistent segmentations as output. We evaluate our method and several baselines on synthetic datasets with multi-view images and multi-granular segmentation, showcasing improved accuracy and viewpoint-consistency.
  arXiv  Detail & Related papers  (2024-05-30T04:14:58Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• Neural Semantic Surface Maps [52.61017226479506]
  We present an automated technique for computing a map between two genus-zero shapes, which matches semantically corresponding regions to one another. Our approach can generate semantic surface-to-surface maps, eliminating manual annotations or any 3D training data requirement.
  arXiv  Detail & Related papers  (2023-09-09T16:21:56Z)
• Explicit3D: Graph Network with Spatial Inference for Single Image 3D Object Detection [35.85544715234846]
  We propose a dynamic sparse graph pipeline named Explicit3D based on object geometry and semantics features. Our experimental results on the SUN RGB-D dataset demonstrate that our Explicit3D achieves better performance balance than the-state-of-the-art.
  arXiv  Detail & Related papers  (2023-02-13T16:19:54Z)
• Learning Implicit Functions for Dense 3D Shape Correspondence of Generic Objects [21.93671761497348]
  A novel implicit function produces a probabilistic embedding to represent each 3D point in a part embedding space. We implement dense correspondence through an inverse function mapping from the part embedding vector to a corresponded 3D point. Our algorithm can automatically generate a confidence score indicating whether there is a correspondence on the target shape.
  arXiv  Detail & Related papers  (2022-12-29T11:57:47Z)
• Point2Seq: Detecting 3D Objects as Sequences [58.63662049729309]
  We present a simple and effective framework, named Point2Seq, for 3D object detection from point clouds. We view each 3D object as a sequence of words and reformulate the 3D object detection task as decoding words from 3D scenes in an auto-regressive manner.
  arXiv  Detail & Related papers  (2022-03-25T00:20:31Z)
• Learning Feature Aggregation for Deep 3D Morphable Models [57.1266963015401]
  We propose an attention based module to learn mapping matrices for better feature aggregation across hierarchical levels. Our experiments show that through the end-to-end training of the mapping matrices, we achieve state-of-the-art results on a variety of 3D shape datasets.
  arXiv  Detail & Related papers  (2021-05-05T16:41:00Z)
• Learning Implicit Functions for Topology-Varying Dense 3D Shape Correspondence [21.93671761497348]
  The goal of this paper is to learn dense 3D shape correspondence for topology-varying objects in an unsupervised manner. Our novel implicit function produces a part embedding vector for each 3D point. We implement dense correspondence through an inverse function mapping from the part embedding to a corresponded 3D point.
  arXiv  Detail & Related papers  (2020-10-23T11:52:06Z)
• Canonical 3D Deformer Maps: Unifying parametric and non-parametric methods for dense weakly-supervised category reconstruction [79.98689027127855]
  We propose a new representation of the 3D shape of common object categories that can be learned from a collection of 2D images of independent objects. Our method builds in a novel way on concepts from parametric deformation models, non-parametric 3D reconstruction, and canonical embeddings. It achieves state-of-the-art results in dense 3D reconstruction on public in-the-wild datasets of faces, cars, and birds.
  arXiv  Detail & Related papers  (2020-08-28T15:44:05Z)
• OS2D: One-Stage One-Shot Object Detection by Matching Anchor Features [14.115782214599015]
  One-shot object detection consists in detecting objects defined by a single demonstration. We build the one-stage system that performs localization and recognition jointly. Experimental evaluation on several challenging domains shows that our method can detect unseen classes.
  arXiv  Detail & Related papers  (2020-03-15T11:39:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.