FSAG: Enhancing Human-to-Dexterous-Hand Finger-Specific Affordance Grounding via Diffusion Models

• URL: http://arxiv.org/abs/2601.08246v2
• Date: Thu, 12 Mar 2026 07:24:53 GMT
• Title: FSAG: Enhancing Human-to-Dexterous-Hand Finger-Specific Affordance Grounding via Diffusion Models
• Abstract summary: Dexterous grasp synthesis must satisfy functional intent and physical feasibility, yet existing pipelines often decouple semantic grounding from refinement.<n>We propose a data-efficient framework that bypasses robot grasp data collection by exploiting object-centric semantic priors in pretrained generative diffusion models.<n>Our results highlight a path toward scalable, hardware-agnostic dexterous manipulation driven by human demonstrations and pretrained generative models.
• Score: 11.581489292735418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dexterous grasp synthesis must jointly satisfy functional intent and physical feasibility, yet existing pipelines often decouple semantic grounding from refinement, yielding unstable or non-functional contacts under object and pose variations. This challenge is exacerbated by the high dimensionality and kinematic diversity of multi-fingered hands, which makes many methods rely on large, hardware-specific grasp datasets collected in simulation or through costly real-world trials. We propose a data-efficient framework that bypasses robot grasp data collection by exploiting object-centric semantic priors in pretrained generative diffusion models. Temporally aligned and fine-grained grasp affordances are extracted from raw human video demonstrations and fused with 3D scene geometry from depth images to infer semantically grounded contact targets. We further incorporate these affordance regions into the grasp refinement objective, explicitly guiding each fingertip toward its predicted region during optimization. The resulting system produces stable, human-intuitive multi-contact grasps across common objects and tools, while exhibiting strong generalization to previously unseen object instances within a category, pose variations, and multiple hand embodiments.This work (i) introduces a semantic affordance extraction pipeline leveraging vision--language generative priors for dexterous grasping, (ii) demonstrates cross-hand generalization without constructing hardware-specific grasp datasets, and (iii) establishes that a single depth modality suffices for high-performance grasp synthesis when coupled with foundation-model semantics. Our results highlight a path toward scalable, hardware-agnostic dexterous manipulation driven by human demonstrations and pretrained generative models.

Related papers

• Real-World Cooperative Bimanual Dexterous Grasp of Large Objects from Single-View Observations [18.70687400045831]
  Bimanual dexterous grasping of large objects is a critical challenge in robotic manipulation.<n>We propose a real-world bimanual grasping framework that includes a multimodal dataset capturing joint angles, visual observations and force signals.<n>Our approach enables the synthesis of executable bimanual grasps from single-view inputs, reducing dependence on complete 3D object models.
  arXiv  Detail & Related papers  (2026-08-11T02:24:27Z)
• DCGrasp: Distance-aware Controllable Grasp Generation [64.30284326160826]
  We present DCGrasp, a controllable grasp generation system built on a novel grasp energy term.<n>This term computes Distance Profile, a signed distance from each hand to the nearest object point.<n>Our experiments show that DCGrasp produces high-quality, physically plausible grasps with flexible user control.
  arXiv  Detail & Related papers  (2026-06-29T08:02:02Z)
• DisPOSE: Projected Polystochastic Diffusion for Self-Supervised Multi-View 3D Human Pose Estimation [58.47973015036709]
  DisPOSE is a self-supervised framework that approximates the inherently discrete multi-view person-assignment problem.<n>By employing differentiable Sinkhorn projections during denoising, our model learns to guide solutions toward valid and feasible assignments.<n>The proposed approach outperforms current state-of-the-art self-supervised methods on standard datasets.
  arXiv  Detail & Related papers  (2026-06-05T16:14:39Z)
• HandX: Scaling Bimanual Motion and Interaction Generation [55.486397101415]
  HandX is a unified foundation spanning data, annotation, and evaluation.<n>We consolidate and filter existing datasets for quality, and collect a new motion-capture dataset.<n>Experiments demonstrate high-quality dexterous motion generation, supported by our newly proposed hand-focused metrics.
  arXiv  Detail & Related papers  (2026-03-30T17:59:49Z)
• AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis [50.793806818677716]
  AffordGrasp produces physically stable and semantically faithful human grasps with high precision.<n>AffordGrasp integrates an affordance-aware latent representation of hand poses with a dual-conditioning diffusion process.<n>We evaluate AffordGrasp across four instruction-augmented benchmarks derived from HO-3D, OakInk, GRAB, and AffordPose.
  arXiv  Detail & Related papers  (2026-03-09T06:56:35Z)
• UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling [45.29560152294065]
  UniHand is a unified diffusion-based framework that formulates both estimation and generation as conditional motion synthesis.<n>Visual observations are encoded with a frozen backbone, while a dedicated hand perceptron extracts hand-specific cues directly from image features.<n>A latent diffusion model then synthesizes consistent motion sequences from diverse conditions.
  arXiv  Detail & Related papers  (2026-02-25T06:53:15Z)
• Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations [71.15185260689928]
  We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations.<n>Our approach makes two key contributions: (1) HOP, a Hand-Object Planner, which can synthesize diverse hand-object trajectories; and (2) HOT, a Hand-Object Tracker, that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning.
  arXiv  Detail & Related papers  (2025-12-22T17:08:54Z)
• S3OD: Towards Generalizable Salient Object Detection with Synthetic Data [38.10559747985342]
  We introduce S3OD, a dataset of over 139,000 high-resolution images created through our multi-modal diffusion pipeline.<n>We propose a streamlined multi-mask decoder that naturally handles the inherent ambiguity in salient object detection.
  arXiv  Detail & Related papers  (2025-10-24T16:10:09Z)
• MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation [28.75149480374178]
  MEgoHand is a framework that synthesizes physically plausible hand-object interactions from egocentric RGB, text, and initial hand pose.<n>It achieves substantial reductions in wrist translation error and joint rotation error, highlighting its capacity to accurately model fine-grained hand joint structures.
  arXiv  Detail & Related papers  (2025-05-22T12:37:47Z)
• SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories [124.24041272390954]
  Modeling hand-object interaction priors holds significant potential to advance robotic and embodied AI systems.<n>We introduce SIGHT, a novel task focused on generating realistic and physically plausible 3D hand-object interaction trajectories from a single image.<n>We propose SIGHT-Fusion, a novel diffusion-based image-text conditioned generative model that tackles this task by retrieving the most similar 3D object mesh from a database.
  arXiv  Detail & Related papers  (2025-03-28T20:53:20Z)
• G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation [65.86819811007157]
  We present G3Flow, a novel framework that constructs real-time semantic flow, a dynamic, object-centric 3D representation by leveraging foundation models.<n>Our approach uniquely combines 3D generative models for digital twin creation, vision foundation models for semantic feature extraction, and robust pose tracking for continuous semantic flow updates.<n>Our results demonstrate the effectiveness of G3Flow in enhancing real-time dynamic semantic feature understanding for robotic manipulation policies.
  arXiv  Detail & Related papers  (2024-11-27T14:17:43Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.