SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation

• URL: http://arxiv.org/abs/2311.15707v2
• Date: Wed, 6 Mar 2024 12:23:29 GMT
• Title: SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation
• Abstract summary: We introduce SAM-6D, a novel framework designed to realize the task through two steps, including instance segmentation and pose estimation. Given the target objects, SAM-6D employs two dedicated sub-networks, namely Instance Model (ISM) and Pose Estimation Model (PEM) ISM takes SAM as an advanced starting point to generate all possible object proposals and selectively preserves valid ones. PEM performs a two-stage point matching process featuring a novel design of background tokens to construct dense 3D-3D correspondence.
• Score: 45.951483980557825
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Zero-shot 6D object pose estimation involves the detection of novel objects with their 6D poses in cluttered scenes, presenting significant challenges for model generalizability. Fortunately, the recent Segment Anything Model (SAM) has showcased remarkable zero-shot transfer performance, which provides a promising solution to tackle this task. Motivated by this, we introduce SAM-6D, a novel framework designed to realize the task through two steps, including instance segmentation and pose estimation. Given the target objects, SAM-6D employs two dedicated sub-networks, namely Instance Segmentation Model (ISM) and Pose Estimation Model (PEM), to perform these steps on cluttered RGB-D images. ISM takes SAM as an advanced starting point to generate all possible object proposals and selectively preserves valid ones through meticulously crafted object matching scores in terms of semantics, appearance and geometry. By treating pose estimation as a partial-to-partial point matching problem, PEM performs a two-stage point matching process featuring a novel design of background tokens to construct dense 3D-3D correspondence, ultimately yielding the pose estimates. Without bells and whistles, SAM-6D outperforms the existing methods on the seven core datasets of the BOP Benchmark for both instance segmentation and pose estimation of novel objects.

Related papers

• Prior-free relative 6D pose estimation of multiple object instances [15.516176223256641]
  We introduce prior-free relative 6D pose estimation, which lifts the assumption of knowing which object is to be posed within the scene.<n>This novel setting aims to estimate the relative poses of multiple instances of an unknown object within the same image, without requiring CAD models, templates, or reference images.
  arXiv  Detail & Related papers  (2026-09-08T16:07:33Z)
• Any6D: Model-free 6D Pose Estimation of Novel Objects [76.30057578269668]
  We introduce Any6D, a model-free framework for 6D object pose estimation. It requires only a single RGB-D anchor image to estimate both the 6D pose and size of unknown objects in novel scenes. We evaluate our method on five challenging datasets.
  arXiv  Detail & Related papers  (2025-03-24T13:46:21Z)
• FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects [55.77542145604758]
  FoundationPose is a unified foundation model for 6D object pose estimation and tracking. Our approach can be instantly applied at test-time to a novel object without fine-tuning.
  arXiv  Detail & Related papers  (2023-12-13T18:28:09Z)
• ZS6D: Zero-shot 6D Object Pose Estimation using Vision Transformers [9.899633398596672]
  We introduce ZS6D, for zero-shot novel object 6D pose estimation. Visual descriptors, extracted using pre-trained Vision Transformers (ViT), are used for matching rendered templates. Experiments are performed on LMO, YCBV, and TLESS datasets.
  arXiv  Detail & Related papers  (2023-09-21T11:53:01Z)
• SA6D: Self-Adaptive Few-Shot 6D Pose Estimator for Novel and Occluded Objects [24.360831082478313]
  We propose a few-shot pose estimation (FSPE) approach called SA6D. It uses a self-adaptive segmentation module to identify the novel target object and construct a point cloud model of the target object. We evaluate SA6D on real-world tabletop object datasets and demonstrate that SA6D outperforms existing FSPE methods.
  arXiv  Detail & Related papers  (2023-08-31T08:19:26Z)
• MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare [84.80956484848505]
  MegaPose is a method to estimate the 6D pose of novel objects, that is, objects unseen during training. We present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner.
  arXiv  Detail & Related papers  (2022-12-13T19:30:03Z)
• Unseen Object 6D Pose Estimation: A Benchmark and Baselines [62.8809734237213]
  We propose a new task that enables and facilitates algorithms to estimate the 6D pose estimation of novel objects during testing. We collect a dataset with both real and synthetic images and up to 48 unseen objects in the test set. By training an end-to-end 3D correspondences network, our method finds corresponding points between an unseen object and a partial view RGBD image accurately and efficiently.
  arXiv  Detail & Related papers  (2022-06-23T16:29:53Z)
• Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB Image [27.234658117816103]
  We propose a single-stage, keypoint-based approach for category-level object pose estimation. The proposed network performs 2D object detection, detects 2D keypoints, estimates 6-DoF pose, and regresses relative bounding cuboid dimensions. We conduct extensive experiments on the challenging Objectron benchmark, outperforming state-of-the-art methods on the 3D IoU metric.
  arXiv  Detail & Related papers  (2021-09-13T17:55:00Z)
• Spatial Attention Improves Iterative 6D Object Pose Estimation [52.365075652976735]
  We propose a new method for 6D pose estimation refinement from RGB images. Our main insight is that after the initial pose estimate, it is important to pay attention to distinct spatial features of the object. We experimentally show that this approach learns to attend to salient spatial features and learns to ignore occluded parts of the object, leading to better pose estimation across datasets.
  arXiv  Detail & Related papers  (2021-01-05T17:18:52Z)
• 3D Registration for Self-Occluded Objects in Context [66.41922513553367]
  We introduce the first deep learning framework capable of effectively handling this scenario. Our method consists of an instance segmentation module followed by a pose estimation one. It allows us to perform 3D registration in a one-shot manner, without requiring an expensive iterative procedure.
  arXiv  Detail & Related papers  (2020-11-23T08:05:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.