LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model

• URL: http://arxiv.org/abs/2406.04659v1
• Date: Fri, 7 Jun 2024 05:58:35 GMT
• Title: LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model
• Abstract summary: LocLLM is the first Large-Language Model based keypoint localization model. It takes images and text instructions as inputs and outputs the desired keypoint coordinates. It shows remarkable performance on standard 2D/3D keypoint localization benchmarks.
• Score: 52.35027502274539
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The capacity of existing human keypoint localization models is limited by keypoint priors provided by the training data. To alleviate this restriction and pursue more general model, this work studies keypoint localization from a different perspective by reasoning locations based on keypiont clues in text descriptions. We propose LocLLM, the first Large-Language Model (LLM) based keypoint localization model that takes images and text instructions as inputs and outputs the desired keypoint coordinates. LocLLM leverages the strong reasoning capability of LLM and clues of keypoint type, location, and relationship in textual descriptions for keypoint localization. To effectively tune LocLLM, we construct localization-based instruction conversations to connect keypoint description with corresponding coordinates in input image, and fine-tune the whole model in a parameter-efficient training pipeline. LocLLM shows remarkable performance on standard 2D/3D keypoint localization benchmarks. Moreover, incorporating language clues into the localization makes LocLLM show superior flexibility and generalizable capability in cross dataset keypoint localization, and even detecting novel type of keypoints unseen during training.

Related papers

• Talking Points: Describing and Localizing Pixels [17.428135548304308]
  We introduce a novel framework for pixel level grounding.<n>The framework consists of two complementary components: a Point Descriptor that generates rich, contextual descriptions of individual keypoints, and a Point Localizer that regresses precise pixel coordinates from these descriptions.
  arXiv  Detail & Related papers  (2025-10-16T11:42:03Z)
• PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment [1.820765907065129]
  We propose PoseLLM, the first Large Language Model (LLM)-based pose estimation framework that replaces the linear projector with a nonlinear vision-language connector.<n>Our work demonstrates that a simple yet powerful nonlinear connector significantly boosts localization accuracy without sacrificing generalization.
  arXiv  Detail & Related papers  (2025-07-12T04:53:39Z)
• ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models [57.57832348655715]
  We propose a novel zero-shot approach for keypoint detection on 3D shapes. Our method utilizes the rich knowledge embedded within Multi-Modal Large Language Models.
  arXiv  Detail & Related papers  (2024-12-09T08:31:57Z)
• KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension [31.283133365170052]
  We introduce Semantic Keypoint, which aims to comprehend keypoints across different task scenarios. We also introduce KptLLM, a unified multimodal model that utilizes an identify-then-detect strategy. KptLLM adeptly handles various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations.
  arXiv  Detail & Related papers  (2024-11-04T06:42:24Z)
• Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation [3.976851945232775]
  Current approaches for sign language recognition rely on RGB video inputs, which are vulnerable to fluctuations in the background. We propose a multi-stream keypoint attention network to depict a sequence of keypoints produced by a readily available keypoint estimator. We carry out comprehensive experiments on well-known benchmarks like Phoenix-2014, Phoenix-2014T, and CSL-Daily to showcase the efficacy of our methodology.
  arXiv  Detail & Related papers  (2024-05-09T10:58:37Z)
• PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs [55.8550939439138]
  Vision-Language Models (VLMs) have shown immense potential by integrating large language models with vision systems. These models face challenges in the fundamental computer vision task of object localisation, due to their training on multimodal data containing mostly captions. We introduce an input-agnostic Positional Insert (PIN), a learnable spatial prompt, containing a minimal set of parameters that are slid inside the frozen VLM. Our PIN module is trained with a simple next-token prediction task on synthetic data without requiring the introduction of new output heads.
  arXiv  Detail & Related papers  (2024-02-13T18:39:18Z)
• Localized Symbolic Knowledge Distillation for Visual Commonsense Models [150.18129140140238]
  We build Localized Visual Commonsense models, which allow users to specify (multiple) regions as input. We train our model by sampling localized commonsense knowledge from a large language model. We find that training on the localized commonsense corpus can successfully distill existing vision-language models to support a reference-as-input interface.
  arXiv  Detail & Related papers  (2023-12-08T05:23:50Z)
• Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching [74.75284453828017]
  Open-Vocabulary Keypoint Detection (OVKD) task is innovatively designed to use text prompts for identifying arbitrary keypoints across any species. We have developed a novel framework named Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM) This framework combines vision and language models, creating an interplay between language features and local keypoint visual features.
  arXiv  Detail & Related papers  (2023-10-08T07:42:41Z)
• PointLLM: Empowering Large Language Models to Understand Point Clouds [63.39876878899682]
  PointLLM understands colored object point clouds with human instructions. It generates contextually appropriate responses, illustrating its grasp of point clouds and common sense.
  arXiv  Detail & Related papers  (2023-08-31T17:59:46Z)
• End-to-End Learning of Keypoint Representations for Continuous Control from Images [84.8536730437934]
  We show that it is possible to learn efficient keypoint representations end-to-end, without the need for unsupervised pre-training, decoders, or additional losses. Our proposed architecture consists of a differentiable keypoint extractor that feeds the coordinates directly to a soft actor-critic agent.
  arXiv  Detail & Related papers  (2021-06-15T09:17:06Z)
• UKPGAN: A General Self-Supervised Keypoint Detector [43.35270822722044]
  UKPGAN is a general self-supervised 3D keypoint detector. Our keypoints align well with human annotated keypoint labels. Our model is stable under both rigid and non-rigid transformations.
  arXiv  Detail & Related papers  (2020-11-24T09:08:21Z)
• Learning Local Features with Context Aggregation for Visual Localization [24.167882373322957]
  Keypoint detection and description is fundamental yet important in many vision applications. Most existing methods use detect-then-describe or detect-and-describe strategy to learn local features without considering their context information. In this paper, we focus on the fusion of low-level textual information and high-level semantic context information to improve the discrimitiveness of local features.
  arXiv  Detail & Related papers  (2020-05-26T17:19:06Z)
• UR2KiD: Unifying Retrieval, Keypoint Detection, and Keypoint Description without Local Correspondence Supervision [16.68130648568593]
  Three related tasks, namely keypoint detection, description, and image retrieval can be jointly tackled using a single unified framework. By leveraging diverse information from sequential layers of a standard ResNet-based architecture, we are able to extract keypoints and descriptors that encode local information. Global information for image retrieval is encoded in an end-to-end pipeline, based on pooling of the aforementioned local responses.
  arXiv  Detail & Related papers  (2020-01-20T21:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.