FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning

• URL: http://arxiv.org/abs/2607.05780v1
• Date: Tue, 07 Jul 2026 03:10:57 GMT
• Title: FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning
• Abstract summary: We show that robots trained on specific tools fail to transfer the same function to novel ones.<n>We propose FunctiOnal Reasoning and Grounded Execution (FORGE), a policy that decouples functional reasoning from action execution.<n>FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world.
• Score: 36.557151261284595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones -- a gap we formalize as functional generalization. Such tools share a common functional intent that is visually recognizable, yet this perceptual similarity does not carry over to action space, where each tool demands an entirely different motor pattern. To bridge this gap, we explore intermediate representations including affordance images, human video prompts, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability. Building on this, we propose FunctiOnal Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution: predicting generalizable keypoint trajectories from action-free data, then grounding them into robot actions with limited demonstrations. On a seven-tool hitting-function benchmark, FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world, achieving over 2X improvement in average success rate.

Related papers

• CORE: Common Outcome Regularities from Action-Free Visual Demonstrations for Robot Manipulation [26.48033419994886]
  CORE is a policy learning framework that extracts Common Outcome Regularities from visual demonstrations.<n> CORE exploits a key observation: although successful trajectories for the same task can be diverse, their terminal states often share stable object configurations.
  arXiv  Detail & Related papers  (2026-06-28T17:27:23Z)
• Guava: An Effective and Universal Harness for Embodied Manipulation [74.34187069605844]
  We develop an end-to-end training pipeline that distills embodied manipulation capabilities into a 4B open-source model.<n>Results suggest that a well-designed harness can serve as a scalable, model-agnostic interface for embodied manipulation.
  arXiv  Detail & Related papers  (2026-06-16T18:09:26Z)
• AFUN: Towards an Affordance Foundation Model for Functionality Understanding [12.890216832485647]
  We present ourmodel, a step towards an affordance foundation model for functionality understanding.<n>We build a large-scale standardized data pipeline that converts heterogeneous robot, human, simulation, and real-world scan data into a shared affordance schema.<n>Ourmodel outperforms all baselines by a large margin across 8 test sets from 4 benchmarks.
  arXiv  Detail & Related papers  (2026-06-01T17:50:16Z)
• ArtiSG: Functional 3D Scene Graph Construction via Human-demonstrated Articulated Objects Manipulation [51.54082859171464]
  ArtiSG is a framework that constructs functional 3D scene graphs by encoding human demonstrations into structured robotic memory.<n>We show that ArtiSG significantly outperforms baselines in functional element recall and articulation estimation precision.
  arXiv  Detail & Related papers  (2025-12-31T13:10:40Z)
• Adaptive Inverse Kinematics Framework for Learning Variable-Length Tool Manipulation in Robotics [0.0]
  Conventional robots possess a limited understanding of their kinematics and are confined to preprogrammed tasks.<n>We introduce a pioneering framework to expand the capabilities of the robot's inverse kinematics solver.<n>Our model achieves virtually indistinguishable performance when employing two distinct tools of different lengths.
  arXiv  Detail & Related papers  (2025-10-30T14:44:24Z)
• FUNCanon: Learning Pose-Aware Action Primitives via Functional Object Canonicalization for Generalizable Robotic Manipulation [25.631729484747087]
  We introduce FunCanon, a framework that converts long-horizon manipulation tasks into sequences of action chunks.<n>These chunks focus policy learning on the actions themselves, rather than isolated tasks.<n>Experiments on simulated and real-world benchmarks demonstrate category-level generalization, cross-task behavior reuse, and robust sim2real deployment.
  arXiv  Detail & Related papers  (2025-09-23T14:49:05Z)
• SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories [124.24041272390954]
  Modeling hand-object interaction priors holds significant potential to advance robotic and embodied AI systems.<n>We introduce SIGHT, a novel task focused on generating realistic and physically plausible 3D hand-object interaction trajectories from a single image.<n>We propose SIGHT-Fusion, a novel diffusion-based image-text conditioned generative model that tackles this task by retrieving the most similar 3D object mesh from a database.
  arXiv  Detail & Related papers  (2025-03-28T20:53:20Z)
• Learning Granularity-Aware Affordances from Human-Object Interaction for Tool-Based Functional Dexterous Grasping [27.124273762587848]
  Affordance features of objects serve as a bridge in the functional interaction between agents and objects.<n>We propose a granularity-aware affordance feature extraction method for locating functional affordance areas.<n>We use highly activated coarse-grained affordance features in hand-object interaction regions to predict grasp gestures.<n>This forms GAAF-Dex, a complete framework that learns Granularity-Aware Affordances from human-object interaction.
  arXiv  Detail & Related papers  (2024-06-30T07:42:57Z)
• Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation [65.46610405509338]
  We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal. We show that this approach of combining scalably learned track prediction with a residual policy enables diverse generalizable robot manipulation.
  arXiv  Detail & Related papers  (2024-05-02T17:56:55Z)
• Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations [77.31328397965653]
  We introduce Ag2Manip (Agent-Agnostic representations for Manipulation), a framework aimed at surmounting challenges through two key innovations. A novel agent-agnostic visual representation derived from human manipulation videos, with the specifics of embodiments obscured to enhance generalizability. An agent-agnostic action representation abstracting a robot's kinematics to a universal agent proxy, emphasizing crucial interactions between end-effector and object.
  arXiv  Detail & Related papers  (2024-04-26T16:40:17Z)
• Model-Based Visual Planning with Self-Supervised Functional Distances [104.83979811803466]
  We present a self-supervised method for model-based visual goal reaching. Our approach learns entirely using offline, unlabeled data. We find that this approach substantially outperforms both model-free and model-based prior methods.
  arXiv  Detail & Related papers  (2020-12-30T23:59:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.