KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation

• URL: http://arxiv.org/abs/2403.10099v2
• Date: Wed, 20 Mar 2024 07:12:12 GMT
• Title: KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation
• Authors: Ruida Zhang, Chenyangguang Zhang, Yan Di, Fabian Manhardt, Xingyu Liu, Federico Tombari, Xiangyang Ji,
• Abstract summary: KP-RED is a unified KeyPoint-driven REtrieval and Deformation framework. It takes object scans as input and jointly retrieves and deforms the most geometrically similar CAD models.
• Score: 87.23575166061413
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we present KP-RED, a unified KeyPoint-driven REtrieval and Deformation framework that takes object scans as input and jointly retrieves and deforms the most geometrically similar CAD models from a pre-processed database to tightly match the target. Unlike existing dense matching based methods that typically struggle with noisy partial scans, we propose to leverage category-consistent sparse keypoints to naturally handle both full and partial object scans. Specifically, we first employ a lightweight retrieval module to establish a keypoint-based embedding space, measuring the similarity among objects by dynamically aggregating deformation-aware local-global features around extracted keypoints. Objects that are close in the embedding space are considered similar in geometry. Then we introduce the neural cage-based deformation module that estimates the influence vector of each keypoint upon cage vertices inside its local support region to control the deformation of the retrieved shape. Extensive experiments on the synthetic dataset PartNet and the real-world dataset Scan2CAD demonstrate that KP-RED surpasses existing state-of-the-art approaches by a large margin. Codes and trained models will be released in https://github.com/lolrudy/KP-RED.

Related papers

• ShapeMatcher: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation [47.94499636697971]
  We present ShapeMatcher, a unified self-supervised learning framework for joint shape canonicalization, segmentation, retrieval and deformation. The key insight of ShapeMaker is the simultaneous training of the four highly-associated processes: canonicalization, segmentation, retrieval, and deformation.
  arXiv  Detail & Related papers  (2023-11-18T15:44:57Z)
• U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds [84.32525852378525]
  We propose U-RED, an Unsupervised shape REtrieval and Deformation pipeline. It takes an arbitrary object observation as input, typically captured by RGB images or scans, and jointly retrieves and deforms the geometrically similar CAD models. We show that U-RED surpasses existing state-of-the-art approaches by 47.3%, 16.7% and 31.6% respectively under Chamfer Distance.
  arXiv  Detail & Related papers  (2023-08-11T20:56:05Z)
• Enhancing Deformable Local Features by Jointly Learning to Detect and Describe Keypoints [8.390939268280235]
  Local feature extraction is a standard approach in computer vision for tackling important tasks such as image matching and retrieval. We propose DALF, a novel deformation-aware network for jointly detecting and describing keypoints. Our approach also enhances the performance of two real-world applications: deformable object retrieval and non-rigid 3D surface registration.
  arXiv  Detail & Related papers  (2023-04-02T18:01:51Z)
• Robust Change Detection Based on Neural Descriptor Fields [53.111397800478294]
  We develop an object-level online change detection approach that is robust to partially overlapping observations and noisy localization results. By associating objects via shape code similarity and comparing local object-neighbor spatial layout, our proposed approach demonstrates robustness to low observation overlap and localization noises.
  arXiv  Detail & Related papers  (2022-08-01T17:45:36Z)
• UKPGAN: A General Self-Supervised Keypoint Detector [43.35270822722044]
  UKPGAN is a general self-supervised 3D keypoint detector. Our keypoints align well with human annotated keypoint labels. Our model is stable under both rigid and non-rigid transformations.
  arXiv  Detail & Related papers  (2020-11-24T09:08:21Z)
• Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training [3.8073142980733]
  We propose a novel framework for monocular 3D objects detection using only RGB images, called KM3D-Net. We design a fully convolutional model to predict object keypoints, dimension, and orientation, and then combine these estimations with perspective geometry constraints to compute position attribute.
  arXiv  Detail & Related papers  (2020-09-02T00:51:51Z)
• Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation [62.618227434286]
  We present a novel learning approach to recover the 6D poses and sizes of unseen object instances from an RGB-D image. We propose a deep network to reconstruct the 3D object model by explicitly modeling the deformation from a pre-learned categorical shape prior.
  arXiv  Detail & Related papers  (2020-07-16T16:45:05Z)
• Cylindrical Convolutional Networks for Joint Object Detection and Viewpoint Estimation [76.21696417873311]
  We introduce a learnable module, cylindrical convolutional networks (CCNs), that exploit cylindrical representation of a convolutional kernel defined in the 3D space. CCNs extract a view-specific feature through a view-specific convolutional kernel to predict object category scores at each viewpoint. Our experiments demonstrate the effectiveness of the cylindrical convolutional networks on joint object detection and viewpoint estimation.
  arXiv  Detail & Related papers  (2020-03-25T10:24:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.