PhysGraph: Physically-Grounded Graph-Transformer Policies for Bimanual Dexterous Hand-Tool-Object Manipulation

• URL: http://arxiv.org/abs/2603.01436v1
• Date: Mon, 02 Mar 2026 04:32:20 GMT
• Title: PhysGraph: Physically-Grounded Graph-Transformer Policies for Bimanual Dexterous Hand-Tool-Object Manipulation
• Abstract summary: Existing methods naively represent the entire system state as a single configuration vector.<n>We present PhysGraph, a physically-grounded graph transformer policy for challenging bimanual hand-tool-object manipulation.
• Score: 15.661420962372441
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Bimanual dexterous manipulation for tool use remains a formidable challenge in robotics due to the high-dimensional state space and complicated contact dynamics. Existing methods naively represent the entire system state as a single configuration vector, disregarding the rich structural and topological information inherent to articulated hands. We present PhysGraph, a physically-grounded graph transformer policy designed explicitly for challenging bimanual hand-tool-object manipulation. Unlike prior works, we represent the bimanual system as a kinematic graph and introduce per-link tokenization to preserve fine-grained local state information. We propose a physically-grounded bias generator that injects structural priors directly into the attention mechanism, including kinematic spatial distance, dynamic contact states, geometric proximity, and anatomical properties. This allows the policy to explicitly reason about physical interactions rather than learning them implicitly from sparse rewards. Extensive experiments show that PhysGraph significantly outperforms baseline - ManipTrans in manipulation precision and task success rates while using only 51% of the parameters of ManipTrans. Furthermore, the inherent topological flexibility of our architecture shows qualitative zero-shot transfer to unseen tool/object geometries, and is sufficiently general to be trained on three robotic hands (Shadow, Allegro, Inspire).

Related papers

• PhysiFormer: Learning to Simulate Mechanics in World Space [56.16644631495]
  We present PhysiFormer, a diffusion transformer for physically-plausible 3D object motion.<n>It represents objects as 3D meshes expressed in world coordinates.<n>It substantially outperforms autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum-based physical consistency.
  arXiv  Detail & Related papers  (2026-06-25T17:59:00Z)
• RelAfford6D: Relational 6D Affordance Graphs for Constraint-Driven Robotic Manipulation [15.618446008590409]
  We introduce RelAfford6, a training-free framework for complex articulated objects.<n>Our system deduces a semantic topology linking a primary interacting part to its physical anchor.<n>We analytically formulate downstream execution as a satisfaction problem.
  arXiv  Detail & Related papers  (2026-06-25T13:42:26Z)
• Mana: Dexterous Manipulation of Articulated Tools [64.16038120159688]
  Mana is a general sim-to-real framework that reinterprets dexterous manipulation as an animation problem.<n>Inspired by computer animation, Mana employs a coarse-to-fine pipeline that transforms procedurally-generated grasps into manipulation trajectories.<n>Mana achieves zero-shot sim-to-real transfer for both grasping and in-hand manipulation, demonstrating a scalable approach to dexterous articulated tool use.
  arXiv  Detail & Related papers  (2026-06-11T17:59:49Z)
• Learning Equivariant Neural-Augmented Object Dynamics From Few Interactions [9.357701422375824]
  A popular approach is to model objects as sets of 3D particles and learn their motion using graph neural networks.<n>We introduce PIEGraph, a novel approach to combining analytical physics and data-driven models.<n>We show that our method enables accurate dynamics prediction and reliable downstream robotic manipulation planning.
  arXiv  Detail & Related papers  (2026-05-04T15:11:22Z)
• UniHM: Unified Dexterous Hand Manipulation with Vision Language Model [39.2419824041854]
  Planning physically feasible dexterous hand manipulation is a central challenge in robotic manipulation and Embodied AI.<n>We introduce UniHM, the first framework for unified dexterous hand manipulation guided by free-form language commands.
  arXiv  Detail & Related papers  (2026-02-28T16:37:11Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• Grasp-and-Lift: Executable 3D Hand-Object Interaction Reconstruction via Physics-in-the-Loop Optimization [5.146238489510258]
  Dexterous hand manipulation increasingly relies on datasets with precise hand-object trajectory data.<n>Existing resources such as DexYCB and HO3D are primarily optimized for visual alignment but often yield physically implausible interactions when replayed in physics simulators.<n>We propose a simulation-in-the-loop refinement framework that converts these visually aligned trajectories into physically valid ones.
  arXiv  Detail & Related papers  (2026-01-26T04:08:30Z)
• ArtiSG: Functional 3D Scene Graph Construction via Human-demonstrated Articulated Objects Manipulation [51.54082859171464]
  ArtiSG is a framework that constructs functional 3D scene graphs by encoding human demonstrations into structured robotic memory.<n>We show that ArtiSG significantly outperforms baselines in functional element recall and articulation estimation precision.
  arXiv  Detail & Related papers  (2025-12-31T13:10:40Z)
• SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories [124.24041272390954]
  Modeling hand-object interaction priors holds significant potential to advance robotic and embodied AI systems.<n>We introduce SIGHT, a novel task focused on generating realistic and physically plausible 3D hand-object interaction trajectories from a single image.<n>We propose SIGHT-Fusion, a novel diffusion-based image-text conditioned generative model that tackles this task by retrieving the most similar 3D object mesh from a database.
  arXiv  Detail & Related papers  (2025-03-28T20:53:20Z)
• LLM-Based Multi-Agent Systems are Scalable Graph Generative Models [73.28294528654885]
  GraphAgent-Generator (GAG) is a novel simulation-based framework for dynamic, text-attributed social graph generation.<n>GAG simulates the temporal node and edge generation processes for zero-shot social graph generation.<n>The resulting graphs exhibit adherence to seven key macroscopic network properties, achieving an 11% improvement in microscopic graph structure metrics.
  arXiv  Detail & Related papers  (2024-10-13T12:57:08Z)
• One-shot Video Imitation via Parameterized Symbolic Abstraction Graphs [8.872100864022675]
  We propose to interpret video demonstrations through Symbolicized Abstraction Graphs (PSAG) We further ground denote geometric constraints through simulation to estimate non-geometric, visually imperceptible attributes. Our approach has been validated across a range of tasks, such as Cutting Avocado, Cutting Vegetable, Pouring Liquid, Rolling Dough, and Slicing Pizza.
  arXiv  Detail & Related papers  (2024-08-22T18:26:47Z)
• Physics-Encoded Graph Neural Networks for Deformation Prediction under Contact [87.69278096528156]
  In robotics, it's crucial to understand object deformation during tactile interactions. We introduce a method using Physics-Encoded Graph Neural Networks (GNNs) for such predictions. We've made our code and dataset public to advance research in robotic simulation and grasping.
  arXiv  Detail & Related papers  (2024-02-05T19:21:52Z)
• "What's This?" -- Learning to Segment Unknown Objects from Manipulation Sequences [27.915309216800125]
  We present a novel framework for self-supervised grasped object segmentation with a robotic manipulator. We propose a single, end-to-end trainable architecture which jointly incorporates motion cues and semantic knowledge. Our method neither depends on any visual registration of a kinematic robot or 3D object models, nor on precise hand-eye calibration or any additional sensor data.
  arXiv  Detail & Related papers  (2020-11-06T10:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.