Multi-Keypoint Affordance Representation for Functional Dexterous Grasping

• URL: http://arxiv.org/abs/2502.20018v1
• Date: Thu, 27 Feb 2025 11:54:53 GMT
• Title: Multi-Keypoint Affordance Representation for Functional Dexterous Grasping
• Authors: Fan Yang, Dongsheng Luo, Wenrui Chen, Jiacheng Lin, Junjie Cai, Kailun Yang, Zhiyong Li, Yaonan Wang,
• Abstract summary: We propose a multi-keypoint affordance representation for functional dexterous grasping.<n>Our method encodes task-driven grasp configurations by localizing functional contact points.<n>Our method significantly improves affordance localization accuracy, grasp consistency, and generalization to unseen tools and tasks.
• Score: 26.961157077703756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Functional dexterous grasping requires precise hand-object interaction, going beyond simple gripping. Existing affordance-based methods primarily predict coarse interaction regions and cannot directly constrain the grasping posture, leading to a disconnection between visual perception and manipulation. To address this issue, we propose a multi-keypoint affordance representation for functional dexterous grasping, which directly encodes task-driven grasp configurations by localizing functional contact points. Our method introduces Contact-guided Multi-Keypoint Affordance (CMKA), leveraging human grasping experience images for weak supervision combined with Large Vision Models for fine affordance feature extraction, achieving generalization while avoiding manual keypoint annotations. Additionally, we present a Keypoint-based Grasp matrix Transformation (KGT) method, ensuring spatial consistency between hand keypoints and object contact points, thus providing a direct link between visual perception and dexterous grasping actions. Experiments on public real-world FAH datasets, IsaacGym simulation, and challenging robotic tasks demonstrate that our method significantly improves affordance localization accuracy, grasp consistency, and generalization to unseen tools and tasks, bridging the gap between visual affordance learning and dexterous robotic manipulation. The source code and demo videos will be publicly available at https://github.com/PopeyePxx/MKA.

Related papers

• SIGHT: Single-Image Conditioned Generation of Hand Trajectories for Hand-Object Interaction [86.54738165527502]
  We introduce a novel task of generating realistic and diverse 3D hand trajectories given a single image of an object. Hand-object interaction trajectory priors can greatly benefit applications in robotics, embodied AI, augmented reality and related fields.
  arXiv  Detail & Related papers  (2025-03-28T20:53:20Z)
• Articulated Object Manipulation using Online Axis Estimation with SAM2-Based Tracking [59.87033229815062]
  Articulated object manipulation requires precise object interaction, where the object's axis must be carefully considered. Previous research employed interactive perception for manipulating articulated objects, but typically, open-loop approaches often suffer from overlooking the interaction dynamics. We present a closed-loop pipeline integrating interactive perception with online axis estimation from segmented 3D point clouds.
  arXiv  Detail & Related papers  (2024-09-24T17:59:56Z)
• Learning Granularity-Aware Affordances from Human-Object Interaction for Tool-Based Functional Grasping in Dexterous Robotics [27.124273762587848]
  Affordance features of objects serve as a bridge in the functional interaction between agents and objects. We propose a granularity-aware affordance feature extraction method for locating functional affordance areas. We also use highly activated coarse-grained affordance features in hand-object interaction regions to predict grasp gestures. This forms a complete dexterous robotic functional grasping framework GAAF-Dex.
  arXiv  Detail & Related papers  (2024-06-30T07:42:57Z)
• Learning Manipulation by Predicting Interaction [85.57297574510507]
  We propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction. The experimental results demonstrate that MPI exhibits remarkable improvement by 10% to 64% compared with previous state-of-the-art in real-world robot platforms.
  arXiv  Detail & Related papers  (2024-06-01T13:28:31Z)
• GEARS: Local Geometry-aware Hand-object Interaction Synthesis [38.75942505771009]
  We introduce a novel joint-centered sensor designed to reason about local object geometry near potential interaction regions. As an important step towards mitigating the learning complexity, we transform the points from global frame to template hand frame and use a shared module to process sensor features of each individual joint. This is followed by a perceptual-temporal transformer network aimed at capturing correlation among the joints in different dimensions.
  arXiv  Detail & Related papers  (2024-04-02T09:18:52Z)
• Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection [70.96299509159981]
  Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction. Traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner.
  arXiv  Detail & Related papers  (2023-12-04T08:02:59Z)
• Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching [74.75284453828017]
  Open-Vocabulary Keypoint Detection (OVKD) task is innovatively designed to use text prompts for identifying arbitrary keypoints across any species. We have developed a novel framework named Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM) This framework combines vision and language models, creating an interplay between language features and local keypoint visual features.
  arXiv  Detail & Related papers  (2023-10-08T07:42:41Z)
• Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection [81.32280287658486]
  We propose a novel one-stage method, namely Glance and Gaze Network (GGNet) GGNet adaptively models a set of actionaware points (ActPoints) via glance and gaze steps. We design an actionaware approach that effectively matches each detected interaction with its associated human-object pair.
  arXiv  Detail & Related papers  (2021-04-12T08:01:04Z)
• Mutual Graph Learning for Camouflaged Object Detection [31.422775969808434]
  A major challenge is that intrinsic similarities between foreground objects and background surroundings make the features extracted by deep model indistinguishable. We design a novel Mutual Graph Learning model, which generalizes the idea of conventional mutual learning from regular grids to the graph domain. In contrast to most mutual learning approaches that use a shared function to model all between-task interactions, MGL is equipped with typed functions for handling different complementary relations.
  arXiv  Detail & Related papers  (2021-04-03T10:14:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.