RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation

• URL: http://arxiv.org/abs/2603.07691v1
• Date: Sun, 08 Mar 2026 15:46:04 GMT
• Title: RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation
• Abstract summary: RoboPCA is a pose-centered affordance prediction framework that jointly predicts task-appropriate contact regions and poses conditioned on instructions.<n>Human2Afford is a data curation pipeline that automatically recovers scene-level 3D information and infers pose-centered affordance annotations from human demonstrations.<n>RoboPCA outperforms baseline methods on image datasets, simulation, and real robots, and exhibits strong generalization across tasks and categories.
• Score: 35.68205801897266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding spatial affordances -- comprising the contact regions of object interaction and the corresponding contact poses -- is essential for robots to effectively manipulate objects and accomplish diverse tasks. However, existing spatial affordance prediction methods mainly focus on locating the contact regions while delegating the pose to independent pose estimation approaches, which can lead to task failures due to inconsistencies between predicted contact regions and candidate poses. In this work, we propose RoboPCA, a pose-centered affordance prediction framework that jointly predicts task-appropriate contact regions and poses conditioned on instructions. To enable scalable data collection for pose-centered affordance learning, we devise Human2Afford, a data curation pipeline that automatically recovers scene-level 3D information and infers pose-centered affordance annotations from human demonstrations. With Human2Afford, scene depth and the interaction object's mask are extracted to provide 3D context and object localization, while pose-centered affordance annotations are obtained by tracking object points within the contact region and analyzing hand-object interaction patterns to establish a mapping from the 3D hand mesh to the robot end-effector orientation. By integrating geometry-appearance cues through an RGB-D encoder and incorporating mask-enhanced features to emphasize task-relevant object regions into the diffusion-based framework, RoboPCA outperforms baseline methods on image datasets, simulation, and real robots, and exhibits strong generalization across tasks and categories.

Related papers

• Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning [22.1291953459429]
  Bimanual manipulation requires reasoning about where to interact with an object and which arm should perform each action.<n>Existing approaches either treat affordance prediction as coarse part segmentation or rely on geometrics for arm assignment.<n>We propose a hierarchical framework for task-aware bimanual affordance prediction.
  arXiv  Detail & Related papers  (2026-04-09T19:39:45Z)
• HOComp: Interaction-Aware Human-Object Composition [62.93211305213214]
  HOComp is a novel approach for compositing a foreground object onto a human-centric background image.<n> Experimental results on our dataset show that HOComp effectively generates human-object interactions with consistent appearances.
  arXiv  Detail & Related papers  (2025-07-22T17:59:21Z)
• Occlusion-Aware 3D Hand-Object Pose Estimation with Masked AutoEncoders [29.274913619777088]
  We propose an occlusion-aware hand-object pose estimation method based on masked autoencoders, termed as HOMAE.<n>We integrate multi-scale features extracted from the decoder to predict a signed distance field (SDF), capturing both global context and fine-grained geometry.<n>Experiments on challenging DexYCB and HO3Dv2 benchmarks demonstrate that HOMAE achieves state-of-the-art performance in hand-object pose estimation.
  arXiv  Detail & Related papers  (2025-06-12T15:30:47Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Quater-GCN: Enhancing 3D Human Pose Estimation with Orientation and Semi-supervised Training [11.4616369582769]
  We introduce Quater-GCN, a directed graph convolutional network tailored to enhance pose estimation by orientation. Q-GCN excels by not only capturing the spatial dependencies among node joints through their coordinates but also integrating the dynamic context of bone rotations in 2D space. We complement our model with a semi-supervised training strategy that leverages unlabeled data, addressing the challenge of limited orientation ground truth data.
  arXiv  Detail & Related papers  (2024-04-30T06:02:59Z)
• Towards Precise 3D Human Pose Estimation with Multi-Perspective Spatial-Temporal Relational Transformers [28.38686299271394]
  We propose a framework for 3D sequence-to-sequence (seq2seq) human pose detection. Firstly, the spatial module represents the human pose feature by intra-image content, while the frame-image relation module extracts temporal relationships. Our method is evaluated on Human3.6M, a popular 3D human pose detection dataset.
  arXiv  Detail & Related papers  (2024-01-30T03:00:25Z)
• Grounding 3D Object Affordance from 2D Interactions in Images [128.6316708679246]
  Grounding 3D object affordance seeks to locate objects' ''action possibilities'' regions in the 3D space. Humans possess the ability to perceive object affordances in the physical world through demonstration images or videos. We devise an Interaction-driven 3D Affordance Grounding Network (IAG), which aligns the region feature of objects from different sources.
  arXiv  Detail & Related papers  (2023-03-18T15:37:35Z)
• Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps [66.24554680709417]
  Knowing the exact 3D location of workers and robots in a collaborative environment enables several real applications. We propose a non-invasive framework based on depth devices and deep neural networks to estimate the 3D pose of robots from an external camera.
  arXiv  Detail & Related papers  (2022-07-06T08:52:12Z)
• Object Manipulation via Visual Target Localization [64.05939029132394]
  Training agents to manipulate objects, poses many challenges. We propose an approach that explores the environment in search for target objects, computes their 3D coordinates once they are located, and then continues to estimate their 3D locations even when the objects are not visible. Our evaluations show a massive 3x improvement in success rate over a model that has access to the same sensory suite.
  arXiv  Detail & Related papers  (2022-03-15T17:59:01Z)
• Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments [81.38641691636847]
  We rethink the problem of scene reconstruction from an embodied agent's perspective. We reconstruct an interactive scene using RGB-D data stream. This reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models.
  arXiv  Detail & Related papers  (2021-03-30T05:56:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.