RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying

• URL: http://arxiv.org/abs/2605.24394v1
• Date: Sat, 23 May 2026 04:38:58 GMT
• Title: RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying
• Abstract summary: Existing robotic knot tying methods rely on precise topological state tracking with ordered keypoints and explicit edge connectivity.<n>We introduce RoboHitch, a novel framework that learns to perform hitch knot tying from human demonstrations using only disordered 3D keypoints and RGB images.
• Score: 4.437059562306781
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robotic manipulation of deformable linear objects (DLOs) presents significant challenges due to complex dynamics and frequent self-occlusions. Existing robotic knot tying methods typically rely on precise topological state tracking with ordered keypoints and explicit edge connectivity. This reliance makes them prone to failures due to tracking drift and topology mismatch caused by repeated bending and crossings during knot formation.To address these limitations, we introduce RoboHitch, a novel framework that learns to perform hitch knot tying from human demonstrations using only disordered 3D keypoints and RGB images. This eliminates the need for explicit topological order, allowing for more flexible manipulation. Our method employs a dynamic Graph Autoencoder to extract geometric features from untracked keypoints, complemented by a Convolutional Autoencoder that captures essential visual context. A bidirectional cross-attention mechanism then fuses these modalities to jointly predict pick and place affordances, facilitating implicit reasoning about the rope's state and enabling knot tying under occlusion.Real-world experiments demonstrate the effectiveness and generalizability of our approach, successfully completing hitch knots in scenarios with self-occlusions.

Related papers

• CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements [63.40365657068892]
  Transferring human hand demonstrations to robotic grippers has recently emerged as a cost-effective solution for robot learning.<n>Motivated by this gap, we adopt an implicit, data-driven approach guided by fine-grained hand-pose motions.<n>In both simulation and real-robot experiments, our framework enables stable and precise hand-to-gripper transfer of complex spatial manipulations.
  arXiv  Detail & Related papers  (2026-09-07T13:47:22Z)
• RelAfford6D: Relational 6D Affordance Graphs for Constraint-Driven Robotic Manipulation [15.618446008590409]
  We introduce RelAfford6, a training-free framework for complex articulated objects.<n>Our system deduces a semantic topology linking a primary interacting part to its physical anchor.<n>We analytically formulate downstream execution as a satisfaction problem.
  arXiv  Detail & Related papers  (2026-06-25T13:42:26Z)
• Visual Geometry Transformer in the Wild: Distractor-Free 3D Reconstruction [77.75911117889915]
  Current end-to-end multi-view 3D reconstruction methods achieve impressive results, but rely on a restrictive static assumption.<n>This reliance on idealized inputs causes even the most advanced methods to fail in real-world settings.<n>We propose Visual Geometry Transformer in the Wild, an end-to-end framework for robust reconstruction from inconsistent views.
  arXiv  Detail & Related papers  (2026-06-22T02:52:22Z)
• Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation [1.873444918172383]
  We present a perception framework integrating a CNN for permutation-invariant keypoint detection and a YOLOv8-OpenCV pipeline to extract grasping points from structural wrinkles.<n>A proposed bimanual algorithm uses this system to stretch fully folded garments via wrinkles, transitioning to keypoint-based ironing once corners emerge.
  arXiv  Detail & Related papers  (2026-06-04T15:30:49Z)
• Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors [51.096845970243855]
  Whole-body Humanoid-Object Interaction (HOI) is bottlenecked by the scarcity of high-fidelity 3D data.<n>We propose Imagine2Real, a zero-shot HOI framework for flexible, geometry-free interaction.
  arXiv  Detail & Related papers  (2026-05-21T10:15:39Z)
• Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces [116.57196064763924]
  We extend the benchmark coverage by introducing dense tabletop objects and explicit multi-level functional relationships.<n>This work introduces critical challenges involving small-scale, dense, and similar instances.<n>We propose an open-vocabulary pipeline based on 2D visual grounding and 3D graph optimization.
  arXiv  Detail & Related papers  (2026-05-15T09:14:50Z)
• Contrastive Multi-Modal Hypergraph Reasoning for 3D Crowd Mesh Recovery [12.116457701406047]
  Multi-person 3D reconstruction is pivotal for real-world interaction analysis.<n>Current approaches rely on single-modality inputs, which inherently lack geometric guidance.<n>We propose Contrastive Multi-modal Hypergraph Reasoning to synergize semantic, geometric, and pose cues for crowd reconstruction.
  arXiv  Detail & Related papers  (2026-04-01T09:39:01Z)
• Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints [87.13154261503168]
  Motion-controllable video generation is crucial for egocentric applications in virtual reality and embodied AI.<n>Existing methods often struggle to achieve 3D-consistent fine-grained hand articulation.<n>We propose a novel framework that generates egocentric videos from a single reference frame.
  arXiv  Detail & Related papers  (2026-03-12T10:02:23Z)
• Multi-Keypoint Affordance Representation for Functional Dexterous Grasping [26.961157077703756]
  We propose a multi-keypoint affordance representation for functional dexterous grasping.<n>Our method encodes task-driven grasp configurations by localizing functional contact points.<n>Our method significantly improves affordance localization accuracy, grasp consistency, and generalization to unseen tools and tasks.
  arXiv  Detail & Related papers  (2025-02-27T11:54:53Z)
• Occlusion Resilient 3D Human Pose Estimation [52.49366182230432]
  Occlusions remain one of the key challenges in 3D body pose estimation from single-camera video sequences. We demonstrate the effectiveness of this approach compared to state-of-the-art techniques that infer poses from single-camera sequences.
  arXiv  Detail & Related papers  (2024-02-16T19:29:43Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• A Distance-Geometric Method for Recovering Robot Joint Angles From an RGB Image [7.971699294672282]
  We present a novel method for retrieving the joint angles of a robot manipulator using only a single RGB image of its current configuration. Our approach, based on a distance-geometric representation of the configuration space, exploits the knowledge of a robot's kinematic model.
  arXiv  Detail & Related papers  (2023-01-05T12:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.