X-Pose: Detecting Any Keypoints

• URL: http://arxiv.org/abs/2310.08530v2
• Date: Wed, 17 Jul 2024 09:25:24 GMT
• Title: X-Pose: Detecting Any Keypoints
• Authors: Jie Yang, Ailing Zeng, Ruimao Zhang, Lei Zhang,
• Abstract summary: X-Pose is a novel framework for multi-object keypoint detection in images. UniKPT is a large-scale dataset of keypoint detection datasets. X-Pose achieves notable improvements over state-of-the-art non-promptable, visual prompt-based, and textual prompt-based methods.
• Score: 28.274913140048003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work aims to address an advanced keypoint detection problem: how to accurately detect any keypoints in complex real-world scenarios, which involves massive, messy, and open-ended objects as well as their associated keypoints definitions. Current high-performance keypoint detectors often fail to tackle this problem due to their two-stage schemes, under-explored prompt designs, and limited training data. To bridge the gap, we propose X-Pose, a novel end-to-end framework with multi-modal (i.e., visual, textual, or their combinations) prompts to detect multi-object keypoints for any articulated (e.g., human and animal), rigid, and soft objects within a given image. Moreover, we introduce a large-scale dataset called UniKPT, which unifies 13 keypoint detection datasets with 338 keypoints across 1,237 categories over 400K instances. Training with UniKPT, X-Pose effectively aligns text-to-keypoint and image-to-keypoint due to the mutual enhancement of multi-modal prompts based on cross-modality contrastive learning. Our experimental results demonstrate that X-Pose achieves notable improvements of 27.7 AP, 6.44 PCK, and 7.0 AP compared to state-of-the-art non-promptable, visual prompt-based, and textual prompt-based methods in each respective fair setting. More importantly, the in-the-wild test demonstrates X-Pose's strong fine-grained keypoint localization and generalization abilities across image styles, object categories, and poses, paving a new path to multi-object keypoint detection in real applications. Our code and dataset are available at https://github.com/IDEA-Research/X-Pose.

Related papers

• OpenKD: Opening Prompt Diversity for Zero- and Few-shot Keypoint Detection [35.57926269889791]
  We open the prompt diversity from three aspects: modality, semantics (seen v.s. unseen), and language. We propose a novel OpenKD model which leverages multimodal prototype set to support both visual and textual prompting.
  arXiv  Detail & Related papers  (2024-09-30T02:58:05Z)
• PoIFusion: Multi-Modal 3D Object Detection via Fusion at Points of Interest [65.48057241587398]
  PoIFusion is a framework to fuse information of RGB images and LiDAR point clouds at the points of interest (PoIs) Our approach maintains the view of each modality and obtains multi-modal features by computation-friendly projection and computation. We conducted extensive experiments on nuScenes and Argoverse2 datasets to evaluate our approach.
  arXiv  Detail & Related papers  (2024-03-14T09:28:12Z)
• Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching [74.75284453828017]
  Open-Vocabulary Keypoint Detection (OVKD) task is innovatively designed to use text prompts for identifying arbitrary keypoints across any species. We have developed a novel framework named Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM) This framework combines vision and language models, creating an interplay between language features and local keypoint visual features.
  arXiv  Detail & Related papers  (2023-10-08T07:42:41Z)
• KeyPosS: Plug-and-Play Facial Landmark Detection through GPS-Inspired True-Range Multilateration [28.96448680048584]
  KeyPoint Positioning System (KeyPosS) is first framework to deduce exact landmark coordinates by triangulating distances between points of interest and anchor points predicted by a fully convolutional network. Experiments on four datasets demonstrate state-of-the-art performance, with KeyPosS outperforming existing methods in low-resolution settings despite minimal computational overhead.
  arXiv  Detail & Related papers  (2023-05-25T19:30:21Z)
• Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation [24.973118696495977]
  This paper presents a novel end-to-end framework withExplicit box Detection for multi-person Pose estimation, called ED-Pose. It unifies the contextual learning between human-level (global) and keypoint-level (local) information. For the first time, as a fully end-to-end framework with a L1 regression loss, ED-Pose surpasses heatmap-based Top-down methods under the same backbone.
  arXiv  Detail & Related papers  (2023-02-03T08:18:34Z)
• Pose for Everything: Towards Category-Agnostic Pose Estimation [93.07415325374761]
  Category-Agnostic Pose Estimation (CAPE) aims to create a pose estimation model capable of detecting the pose of any class of object given only a few samples with keypoint definition. A transformer-based Keypoint Interaction Module (KIM) is proposed to capture both the interactions among different keypoints and the relationship between the support and query images. We also introduce Multi-category Pose (MP-100) dataset, which is a 2D pose dataset of 100 object categories containing over 20K instances and is well-designed for developing CAPE algorithms.
  arXiv  Detail & Related papers  (2022-07-21T09:40:54Z)
• SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object Detection [78.90102636266276]
  We propose a novel set abstraction method named Semantics-Augmented Set Abstraction (SASA) Based on the estimated point-wise foreground scores, we then propose a semantics-guided point sampling algorithm to help retain more important foreground points during down-sampling. In practice, SASA shows to be effective in identifying valuable points related to foreground objects and improving feature learning for point-based 3D detection.
  arXiv  Detail & Related papers  (2022-01-06T08:54:47Z)
• Rethinking Keypoint Representations: Modeling Keypoints and Poses as Objects for Multi-Person Human Pose Estimation [79.78017059539526]
  We propose a new heatmap-free keypoint estimation method in which individual keypoints and sets of spatially related keypoints (i.e., poses) are modeled as objects within a dense single-stage anchor-based detection framework. In experiments, we observe that KAPAO is significantly faster and more accurate than previous methods, which suffer greatly from heatmap post-processing. Our large model, KAPAO-L, achieves an AP of 70.6 on the Microsoft COCO Keypoints validation set without test-time augmentation.
  arXiv  Detail & Related papers  (2021-11-16T15:36:44Z)
• Weakly Supervised Keypoint Discovery [27.750244813890262]
  We propose a method for keypoint discovery from a 2D image using image-level supervision. Motivated by the weakly-supervised learning approach, our method exploits image-level supervision to identify discriminative parts. Our approach achieves state-of-the-art performance for the task of keypoint estimation on the limited supervision scenarios.
  arXiv  Detail & Related papers  (2021-09-28T01:26:53Z)
• DeepDarts: Modeling Keypoints as Objects for Automatic Scorekeeping in Darts using a Single Camera [75.34178733070547]
  Existing multi-camera solutions for automatic scorekeeping in steel-tip darts are very expensive and thus inaccessible to most players. We present a new approach to keypoint detection and apply it to predict dart scores from a single image taken from any camera angle. We develop a deep convolutional neural network around this idea and use it to predict dart locations and dartboard calibration points.
  arXiv  Detail & Related papers  (2021-05-20T16:25:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.