MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images

• URL: http://arxiv.org/abs/2403.01517v2
• Date: Wed, 8 May 2024 11:54:05 GMT
• Title: MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images
• Authors: Junwen Huang, Hao Yu, Kuan-Ting Yu, Nassir Navab, Slobodan Ilic, Benjamin Busam,
• Abstract summary: We propose a Fuse-Describe-Match strategy for 6D pose estimation from RGB-D images. MatchU is a generic approach that fuses 2D texture and 3D geometric cues for 6D pose prediction of unseen objects.
• Score: 57.71600854525037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent learning methods for object pose estimation require resource-intensive training for each individual object instance or category, hampering their scalability in real applications when confronted with previously unseen objects. In this paper, we propose MatchU, a Fuse-Describe-Match strategy for 6D pose estimation from RGB-D images. MatchU is a generic approach that fuses 2D texture and 3D geometric cues for 6D pose prediction of unseen objects. We rely on learning geometric 3D descriptors that are rotation-invariant by design. By encoding pose-agnostic geometry, the learned descriptors naturally generalize to unseen objects and capture symmetries. To tackle ambiguous associations using 3D geometry only, we fuse additional RGB information into our descriptor. This is achieved through a novel attention-based mechanism that fuses cross-modal information, together with a matching loss that leverages the latent space learned from RGB data to guide the descriptor learning process. Extensive experiments reveal the generalizability of both the RGB-D fusion strategy as well as the descriptor efficacy. Benefiting from the novel designs, MatchU surpasses all existing methods by a significant margin in terms of both accuracy and speed, even without the requirement of expensive re-training or rendering.

Related papers

• Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference [62.99706119370521]
  Humans can easily deduce the relative pose of an unseen object, without label/training, given only a single query-reference image pair. We propose a novel 3D generalizable relative pose estimation method by elaborating (i) with a 2.5D shape from an RGB-D reference, (ii) with an off-the-shelf differentiable, and (iii) with semantic cues from a pretrained model like DINOv2.
  arXiv  Detail & Related papers  (2024-06-26T16:01:10Z)
• RDPN6D: Residual-based Dense Point-wise Network for 6Dof Object Pose Estimation Based on RGB-D Images [13.051302134031808]
  We introduce a novel method for calculating the 6DoF pose of an object using a single RGB-D image. Unlike existing methods that either directly predict objects' poses or rely on sparse keypoints for pose recovery, our approach addresses this challenging task using dense correspondence.
  arXiv  Detail & Related papers  (2024-05-14T10:10:45Z)
• GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects [14.598853174946656]
  We present GenFlow, an approach that enables both accuracy and generalization to novel objects. Our method predicts optical flow between the rendered image and the observed image and refines the 6D pose iteratively. It boosts the performance by a constraint of the 3D shape and the generalizable geometric knowledge learned from an end-to-end differentiable system.
  arXiv  Detail & Related papers  (2024-03-18T06:32:23Z)
• Towards Two-view 6D Object Pose Estimation: A Comparative Study on Fusion Strategy [16.65699606802237]
  Current RGB-based 6D object pose estimation methods have achieved noticeable performance on datasets and real world applications. This paper proposes a framework for 6D object pose estimation that learns implicit 3D information from 2 RGB images.
  arXiv  Detail & Related papers  (2022-07-01T08:22:34Z)
• Unseen Object 6D Pose Estimation: A Benchmark and Baselines [62.8809734237213]
  We propose a new task that enables and facilitates algorithms to estimate the 6D pose estimation of novel objects during testing. We collect a dataset with both real and synthetic images and up to 48 unseen objects in the test set. By training an end-to-end 3D correspondences network, our method finds corresponding points between an unseen object and a partial view RGBD image accurately and efficiently.
  arXiv  Detail & Related papers  (2022-06-23T16:29:53Z)
• Coupled Iterative Refinement for 6D Multi-Object Pose Estimation [64.7198752089041]
  Given a set of known 3D objects and an RGB or RGB-D input image, we detect and estimate the 6D pose of each object. Our approach iteratively refines both pose and correspondence in a tightly coupled manner, allowing us to dynamically remove outliers to improve accuracy.
  arXiv  Detail & Related papers  (2022-04-26T18:00:08Z)
• Pose Estimation of Specific Rigid Objects [0.7931904787652707]
  We address the problem of estimating the 6D pose of rigid objects from a single RGB or RGB-D input image. This problem is of great importance to many application fields such as robotic manipulation, augmented reality, and autonomous driving.
  arXiv  Detail & Related papers  (2021-12-30T14:36:47Z)
• FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation [54.666329929930455]
  We present FFB6D, a Bidirectional fusion network designed for 6D pose estimation from a single RGBD image. We learn to combine appearance and geometry information for representation learning as well as output representation selection. Our method outperforms the state-of-the-art by large margins on several benchmarks.
  arXiv  Detail & Related papers  (2021-03-03T08:07:29Z)
• EPOS: Estimating 6D Pose of Objects with Symmetries [57.448933686429825]
  We present a new method for estimating the 6D pose of rigid objects with available 3D models from a single RGB input. An object is represented by compact surface fragments which allow symmetries in a systematic manner. Correspondences between densely sampled pixels and the fragments are predicted using an encoder-decoder network.
  arXiv  Detail & Related papers  (2020-04-01T17:41:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.