GKDT: General Keypoint Detection Transformer

• URL: http://arxiv.org/abs/2607.00752v1
• Date: Wed, 01 Jul 2026 10:37:29 GMT
• Title: GKDT: General Keypoint Detection Transformer
• Abstract summary: We present a large-scale unified keypoint dataset called MegaKPT.<n>The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets.<n>Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection.
• Score: 85.07219285562591
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: With the emergence of various pre-trained vision and language models, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) model to support diverse tasks has become increasingly important in the field. To this end, we firstly present a large-scale unified keypoint dataset called MegaKPT. The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets, and enjoys high-quality unified annotations with keypoint text descriptions. Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection. Our GKDT supports visual prompts, text prompts, or both. To enhance model training, we also propose a suite of useful strategies such as mix-modal prompted training and dynamic importance sampling. By testing over 22 test sets with seen or unseen objects, our single GKDT model shows strong performance and generality in detecting keypoints on broad categories, with most categories over 90\% PCK@0.1 accuracy, offering high practical applicability to real-world problems. The dataset, models, and codes will be released at https://github.com/AlanLuSun/General-Keypoint-Detection.

Related papers

• KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model [31.59640895434506]
  Keypoints, as structure-aware, pixel-level, and compact representations of objects, play a crucial role in applications such as fine-grained image analysis, object retrieval, and behavior recognition.<n>In this paper, we propose KptLLM++, a novel multimodal large language model that specifically designed for generic keypoint comprehension.<n>By unifying keypoint detection across varied contexts, KptLLM++ establishes itself as an advanced interface, fostering more effective human-AI collaboration.
  arXiv  Detail & Related papers  (2025-07-15T08:52:28Z)
• From Dataset to Real-world: General 3D Object Detection via Generalized Cross-domain Few-shot Learning [13.282416396765392]
  We introduce the first generalized cross-domain few-shot (GCFS) task in 3D object detection.<n>Our solution integrates multi-modal fusion and contrastive-enhanced prototype learning within one framework.<n>To effectively capture domain-specific representations for each class from limited target data, we propose a contrastive-enhanced prototype learning.
  arXiv  Detail & Related papers  (2025-03-08T17:05:21Z)
• Keypoint Abstraction using Large Models for Object-Relative Imitation Learning [78.92043196054071]
  Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for essential object capturing features. We propose KALM, a framework that leverages large pre-trained vision-language models to automatically generate task-relevant and cross-instance consistent keypoints.
  arXiv  Detail & Related papers  (2024-10-30T17:37:31Z)
• Plain-Det: A Plain Multi-Dataset Object Detector [22.848784430833835]
  Plain-Det offers flexibility to accommodate new datasets, in performance across diverse datasets, and training efficiency. We conduct extensive experiments on 13 downstream datasets and Plain-Det demonstrates strong generalization capability.
  arXiv  Detail & Related papers  (2024-07-14T05:18:06Z)
• DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection [111.68263493302499]
  We introduce DetCLIPv3, a high-performing detector that excels at both open-vocabulary object detection and hierarchical labels. DetCLIPv3 is characterized by three core designs: 1) Versatile model architecture; 2) High information density data; and 3) Efficient training strategy. DetCLIPv3 demonstrates superior open-vocabulary detection performance, outperforming GLIPv2, GroundingDINO, and DetCLIPv2 by 18.0/19.6/6.6 AP, respectively.
  arXiv  Detail & Related papers  (2024-04-14T11:01:44Z)
• X-Pose: Detecting Any Keypoints [28.274913140048003]
  X-Pose is a novel framework for multi-object keypoint detection in images. UniKPT is a large-scale dataset of keypoint detection datasets. X-Pose achieves notable improvements over state-of-the-art non-promptable, visual prompt-based, and textual prompt-based methods.
  arXiv  Detail & Related papers  (2023-10-12T17:22:58Z)
• Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching [74.75284453828017]
  Open-Vocabulary Keypoint Detection (OVKD) task is innovatively designed to use text prompts for identifying arbitrary keypoints across any species. We have developed a novel framework named Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM) This framework combines vision and language models, creating an interplay between language features and local keypoint visual features.
  arXiv  Detail & Related papers  (2023-10-08T07:42:41Z)
• ViTPose++: Vision Transformer for Generic Body Pose Estimation [70.86760562151163]
  We show the surprisingly good properties of plain vision transformers for body pose estimation from various aspects. ViTPose employs the plain and non-hierarchical vision transformer as an encoder to encode features and a lightweight decoder to decode body keypoints. We empirically demonstrate that the knowledge of large ViTPose models can be easily transferred to small ones via a simple knowledge token.
  arXiv  Detail & Related papers  (2022-12-07T12:33:28Z)
• Pose for Everything: Towards Category-Agnostic Pose Estimation [93.07415325374761]
  Category-Agnostic Pose Estimation (CAPE) aims to create a pose estimation model capable of detecting the pose of any class of object given only a few samples with keypoint definition. A transformer-based Keypoint Interaction Module (KIM) is proposed to capture both the interactions among different keypoints and the relationship between the support and query images. We also introduce Multi-category Pose (MP-100) dataset, which is a 2D pose dataset of 100 object categories containing over 20K instances and is well-designed for developing CAPE algorithms.
  arXiv  Detail & Related papers  (2022-07-21T09:40:54Z)
• Simple Cues Lead to a Strong Multi-Object Tracker [3.7189423451031356]
  We propose a new type of tracking-by-detection (TbD) for Multi-Object Tracking. We show that a combination of our appearance features with a simple motion model leads to strong tracking results. Our tracker generalizes to four public datasets, namely MOT17, MOT20, BDD100k, and DanceTrack, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-06-09T17:55:51Z)
• SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection [9.924083358178239]
  We propose two variants of self-attention for contextual modeling in 3D object detection. We first incorporate the pairwise self-attention mechanism into the current state-of-the-art BEV, voxel and point-based detectors. Next, we propose a self-attention variant that samples a subset of the most representative features by learning deformations over randomly sampled locations.
  arXiv  Detail & Related papers  (2021-01-07T18:30:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.