Open-vocabulary object 6D pose estimation

• URL: http://arxiv.org/abs/2312.00690v4
• Date: Tue, 25 Jun 2024 06:23:56 GMT
• Title: Open-vocabulary object 6D pose estimation
• Authors: Jaime Corsetti, Davide Boscaini, Changjae Oh, Andrea Cavallaro, Fabio Poiesi,
• Abstract summary: We introduce the new setting of open-vocabulary object 6D pose estimation, in which a textual prompt is used to specify the object of interest. To operate in this setting, we introduce a novel approach that leverages a Vision-Language Model to segment the object of interest from the scenes. We validate our approach on a new benchmark based on two popular datasets, REAL275 and Toyota-Light.
• Score: 31.863333447303273
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce the new setting of open-vocabulary object 6D pose estimation, in which a textual prompt is used to specify the object of interest. In contrast to existing approaches, in our setting (i) the object of interest is specified solely through the textual prompt, (ii) no object model (e.g., CAD or video sequence) is required at inference, and (iii) the object is imaged from two RGBD viewpoints of different scenes. To operate in this setting, we introduce a novel approach that leverages a Vision-Language Model to segment the object of interest from the scenes and to estimate its relative 6D pose. The key of our approach is a carefully devised strategy to fuse object-level information provided by the prompt with local image features, resulting in a feature space that can generalize to novel concepts. We validate our approach on a new benchmark based on two popular datasets, REAL275 and Toyota-Light, which collectively encompass 34 object instances appearing in four thousand image pairs. The results demonstrate that our approach outperforms both a well-established hand-crafted method and a recent deep learning-based baseline in estimating the relative 6D pose of objects in different scenes. Code and dataset are available at https://jcorsetti.github.io/oryon.

Related papers

• Generalizable Single-view Object Pose Estimation by Two-side Generating and Matching [19.730504197461144]
  We present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Our method offers generalization to unseen objects without extensive training, operates with a single reference image of the object, and eliminates the need for 3D object models or multiple views of the object.
  arXiv  Detail & Related papers  (2024-11-24T14:31:50Z)
• High-resolution open-vocabulary object 6D pose estimation [30.835921843505123]
  Horyon is an open-vocabulary VLM-based architecture that addresses relative pose estimation between two scenes of an unseen object. We evaluate our model on a benchmark with a large variety of unseen objects across four datasets.
  arXiv  Detail & Related papers  (2024-06-24T07:53:46Z)
• OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimation [56.028185293563325]
  This paper studies a new open-set problem, the open-vocabulary category-level object pose and size estimation. We first introduce OO3D-9D, a large-scale photorealistic dataset for this task. We then propose a framework built on pre-trained DinoV2 and text-to-image stable diffusion models.
  arXiv  Detail & Related papers  (2024-03-19T03:09:24Z)
• MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare [84.80956484848505]
  MegaPose is a method to estimate the 6D pose of novel objects, that is, objects unseen during training. We present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner.
  arXiv  Detail & Related papers  (2022-12-13T19:30:03Z)
• Unseen Object 6D Pose Estimation: A Benchmark and Baselines [62.8809734237213]
  We propose a new task that enables and facilitates algorithms to estimate the 6D pose estimation of novel objects during testing. We collect a dataset with both real and synthetic images and up to 48 unseen objects in the test set. By training an end-to-end 3D correspondences network, our method finds corresponding points between an unseen object and a partial view RGBD image accurately and efficiently.
  arXiv  Detail & Related papers  (2022-06-23T16:29:53Z)
• Semantic keypoint-based pose estimation from single RGB frames [64.80395521735463]
  We present an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. We show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios.
  arXiv  Detail & Related papers  (2022-04-12T15:03:51Z)
• FS6D: Few-Shot 6D Pose Estimation of Novel Objects [116.34922994123973]
  6D object pose estimation networks are limited in their capability to scale to large numbers of object instances. In this work, we study a new open set problem; the few-shot 6D object poses estimation: estimating the 6D pose of an unknown object by a few support views without extra training.
  arXiv  Detail & Related papers  (2022-03-28T10:31:29Z)
• Weakly Supervised Learning of Keypoints for 6D Object Pose Estimation [73.40404343241782]
  We propose a weakly supervised 6D object pose estimation approach based on 2D keypoint detection. Our approach achieves comparable performance with state-of-the-art fully supervised approaches.
  arXiv  Detail & Related papers  (2022-03-07T16:23:47Z)
• Precise Object Placement with Pose Distance Estimations for Different Objects and Grippers [7.883179102580462]
  Our method estimates multiple 6D object poses together with an object class, a pose distance for object pose estimation, and a pose distance from a target pose for object placement. By incorporating model knowledge into the system, our approach has higher success rates for grasping than state-of-the-art model-free approaches.
  arXiv  Detail & Related papers  (2021-10-03T12:18:59Z)
• Novel Object Viewpoint Estimation through Reconstruction Alignment [45.16865218423492]
  We learn a reconstruct and align approach to estimate the viewpoint of a novel object. In particular, we propose learning two networks: the first maps images to a 3D geometry-aware feature bottleneck and is trained via an image-to-image translation loss. At test time, our model finds the relative transformation that best aligns the bottleneck features of our test image to a reference image.
  arXiv  Detail & Related papers  (2020-06-05T17:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.