PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions

• URL: http://arxiv.org/abs/2605.09538v1
• Date: Sun, 10 May 2026 13:51:45 GMT
• Title: PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions
• Abstract summary: PhysHanDI is a framework that enables full 3D reconstruction of both interacting hands and non-rigid objects.<n>Our key idea is to physically simulate object deformations driven by forces induced from densely reconstructed 3D hand motions.<n>In experiments, PhysHanDI outperforms the state-of-the-art baseline across reconstruction and future prediction.
• Score: 38.846581448310275
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: While existing methods for reconstructing hand-object interactions have made impressive progress, they either focus on rigid or part-wise rigid objects-limiting their ability to model real-world objects (e.g., cloth, stuffed animals) that exhibit highly non-rigid deformations-or model deformable objects without full 3D hand reconstruction. To bridge this gap, we present PhysHanDI (Physics-based Reconstruction of Hand and Deformable Object Interactions), a framework that enables full 3D reconstruction of both interacting hands and non-rigid objects. Our key idea is to physically simulate object deformations driven by forces induced from densely reconstructed 3D hand motions, ensuring that the reconstructed object dynamics are both physically plausible and coherent with the interacting hand movements. Furthermore, we demonstrate that such simulation of object deformations can, in turn, refine and improve hand reconstruction via inverse physics. In experiments, PhysHanDI outperforms the state-of-the-art baseline across reconstruction and future prediction.

Related papers

• SAM3D-Phys: Towards Multi-Object Interactive Simulation in Real World [42.46212677643043]
  SAM3D-Phys is a framework that integrates scene reconstruction with generative 3D priors of SAM3D to recover physically simulatable objects.<n>By recovering complete object geometry and restoring its pose and appearance within the scene, SAM3D-Phys produces clean object representations suitable for physics-based simulation.
  arXiv  Detail & Related papers  (2026-05-28T17:07:51Z)
• TelePhysics: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction [51.01447538245441]
  We propose TelePhysics, a training-free framework that converts a single image into a physically consistent and controllable video.<n>By representing the full scene geometry in a unified spatial coordinate system, TelePhysics resolves object penetration and alignment ambiguity.<n> Experimental results demonstrate that TelePhysics substantially outperforms prior methods in physical fidelity, spatial coherence, and controllability.
  arXiv  Detail & Related papers  (2026-05-19T08:16:44Z)
• SOPHY: Learning to Generate Simulation-Ready Objects with Physical Materials [10.156212838002903]
  SOPHY is a generative model for 3D physics-aware shape synthesis.<n>Our method jointly synthesizes shape, texture, and material properties related to physics-grounded dynamics.
  arXiv  Detail & Related papers  (2025-04-17T06:17:24Z)
• PhysPart: Physically Plausible Part Completion for Interactable Objects [28.91080122885566]
  We tackle the problem of physically plausible part completion for interactable objects.<n>We propose a diffusion-based part generation model that utilizes geometric conditioning.<n>We also demonstrate our applications in 3D printing, robot manipulation, and sequential part generation.
  arXiv  Detail & Related papers  (2024-08-25T04:56:09Z)
• Physically Compatible 3D Object Modeling from a Single Image [109.98124149566927]
  We present a framework that transforms single images into 3D physical objects.<n>Our framework embeds physical compatibility into the reconstruction process.<n>It consistently enhances the physical realism of 3D models over existing methods.
  arXiv  Detail & Related papers  (2024-05-30T21:59:29Z)
• Hand-Object Interaction Controller (HOIC): Deep Reinforcement Learning for Reconstructing Interactions with Physics [12.443255379595278]
  We reconstruct hand manipulating motion with a single RGBD camera by a novel deep reinforcement learning method. We propose object compensation control which establishes direct object control to make the network training more stable. By leveraging the compensation force and torque, we seamlessly upgrade the simple point contact model to a more physical-plausible surface contact model.
  arXiv  Detail & Related papers  (2024-05-04T14:32:13Z)
• PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation [62.53760963292465]
  PhysDreamer is a physics-based approach that endows static 3D objects with interactive dynamics. We present our approach on diverse examples of elastic objects and evaluate the realism of the synthesized interactions through a user study.
  arXiv  Detail & Related papers  (2024-04-19T17:41:05Z)
• REACTO: Reconstructing Articulated Objects from a Single Video [64.89760223391573]
  We propose a novel deformation model that enhances the rigidity of each part while maintaining flexible deformation of the joints. Our method outperforms previous works in producing higher-fidelity 3D reconstructions of general articulated objects.
  arXiv  Detail & Related papers  (2024-04-17T08:01:55Z)
• HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video [70.11702620562889]
  HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can disentangled 3D hand and object from 2D images. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings.
  arXiv  Detail & Related papers  (2023-11-30T10:50:35Z)
• ShapeGraFormer: GraFormer-Based Network for Hand-Object Reconstruction from a Single Depth Map [11.874184782686532]
  We propose the first approach for realistic 3D hand-object shape and pose reconstruction from a single depth map. Our pipeline additionally predicts voxelized hand-object shapes, having a one-to-one mapping to the input voxelized depth. In addition, we show the impact of adding another GraFormer component that refines the reconstructed shapes based on the hand-object interactions.
  arXiv  Detail & Related papers  (2023-10-18T09:05:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.