THOM: Generating Physically Plausible Hand-Object Meshes From Text

• URL: http://arxiv.org/abs/2604.02736v2
• Date: Mon, 06 Apr 2026 04:52:22 GMT
• Title: THOM: Generating Physically Plausible Hand-Object Meshes From Text
• Abstract summary: Generation of 3D hand-object interactions (HOIs) from text is crucial for dexterous robotic grasping and VR/AR content generation.<n>We introduce THOM, a training-free framework that generates photorealistic, physically plausible 3D HOI meshes without the need for a template object mesh.<n>THOM consistently surpasses state-of-the-art methods in terms of text alignment, visual realism, and interaction plausibility.
• Score: 44.69476210110844
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The generation of 3D hand-object interactions (HOIs) from text is crucial for dexterous robotic grasping and VR/AR content generation, requiring both high visual fidelity and physical plausibility. Nevertheless, the ill-posed problem of mesh extraction from text-generated Gaussians, and physics-based optimization on the erroneous meshes pose challenges. To address these issues, we introduce THOM, a training-free framework that generates photorealistic, physically plausible 3D HOI meshes without the need for a template object mesh. THOM employs a two-stage pipeline, initially generating the hand and object Gaussians, followed by physics-based HOI optimization. Our new mesh extraction method and vertex-to-Gaussian mapping explicitly assign Gaussian elements to mesh vertices, allowing topology-aware regularization. Furthermore, we improve the physical plausibility of interactions by VLM-guided translation refinement and contact-aware optimization. Comprehensive experiments demonstrate that THOM consistently surpasses state-of-the-art methods in terms of text alignment, visual realism, and interaction plausibility.

Related papers

• G-Skin: Learning to Bind 3D Gaussians with Generative Visual Priors [51.398600453223274]
  G-Skin is a novel generative skinning framework designed for expressive and high-fidelity animation with 3D Gaussian representation.<n>We introduce a skeleton-controllable image generation model leveraging 2D vision foundation models to distill powerful motion priors into pseudo-guidance.<n>G-Skin also generalizes flexibly to the augmented variants of 3D Gaussian representation designed to mitigate animation-induced rendering artifacts.
  arXiv  Detail & Related papers  (2026-08-03T05:45:23Z)
• PhysMirror: Physics-Aware Mirror Object Generation [27.32616510065338]
  We introduce a novel, end-to-end physics-aware generation framework namely PhysMirror.<n>Our method automatically lifts prompted objects into 3D meshes and constructs a lightweight, mathematically exact mirror scene.<n>We also introduce Mirror Consistency Score (MCS), reference-free, fully automated metric that quantifies physical correctness.
  arXiv  Detail & Related papers  (2026-07-03T16:32:02Z)
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding [50.098085774845195]
  We propose a paradigm shift by leveraging the implicit spatial prior within large-scale generation models.<n>We introduce GeneEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator.
  arXiv  Detail & Related papers  (2026-03-19T17:59:58Z)
• High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions [83.73866089033585]
  We propose a novel approach by simultaneously refining both the latent code of a pretrained 3D-Aware GAN model for texture editing and the expression code of the driven 3DMM model for mesh editing.<n>To optimize the Dual Mappers, we propose a Text-Guided Optimization method, leveraging a CLIP-based objective function with expression text prompts as targets.
  arXiv  Detail & Related papers  (2026-03-16T03:25:10Z)
• Grasp-and-Lift: Executable 3D Hand-Object Interaction Reconstruction via Physics-in-the-Loop Optimization [5.146238489510258]
  Dexterous hand manipulation increasingly relies on datasets with precise hand-object trajectory data.<n>Existing resources such as DexYCB and HO3D are primarily optimized for visual alignment but often yield physically implausible interactions when replayed in physics simulators.<n>We propose a simulation-in-the-loop refinement framework that converts these visually aligned trajectories into physically valid ones.
  arXiv  Detail & Related papers  (2026-01-26T04:08:30Z)
• GS-Verse: Mesh-based Gaussian Splatting for Physics-aware Interaction in Virtual Reality [1.2271175670122132]
  Current techniques for physically manipulating 3D content within Virtual Reality (VR) often face significant limitations.<n>We introduce GS-Verse, a novel method designed to overcome these challenges by directly integrating an object's mesh with a Gaussian Splatting representation.<n>Our system is designed to be physics-engine-agnostic, granting developers robust deployment flexibility.
  arXiv  Detail & Related papers  (2025-10-13T19:36:47Z)
• PIG: Physically-based Multi-Material Interaction with 3D Gaussians [14.097146027458368]
  PIG: Physically-Based Multi-Material Interaction with 3D Gaussians is a novel approach that combines 3D object segmentation with the simulation of objects interacting in high precision.<n>We show that our method not only outperforms the state-of-the-art (SOTA) in terms of visual quality, but also opens up new directions and pipelines for the field of physically realistic scene generation.
  arXiv  Detail & Related papers  (2025-06-09T11:25:21Z)
• SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories [124.24041272390954]
  Modeling hand-object interaction priors holds significant potential to advance robotic and embodied AI systems.<n>We introduce SIGHT, a novel task focused on generating realistic and physically plausible 3D hand-object interaction trajectories from a single image.<n>We propose SIGHT-Fusion, a novel diffusion-based image-text conditioned generative model that tackles this task by retrieving the most similar 3D object mesh from a database.
  arXiv  Detail & Related papers  (2025-03-28T20:53:20Z)
• DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation [10.250715657201363]
  We introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.
  arXiv  Detail & Related papers  (2024-10-09T10:41:08Z)
• GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation [60.33467489955188]
  This paper studies the problem of estimating physical properties (system identification) through visual observations. To facilitate geometry-aware guidance in physical property estimation, we introduce a novel hybrid framework. We propose a new dynamic 3D Gaussian framework based on motion factorization to recover the object as 3D Gaussian point sets. In addition to the extracted object surfaces, the Gaussian-informed continuum also enables the rendering of object masks during simulations.
  arXiv  Detail & Related papers  (2024-06-21T07:37:17Z)
• MaGS: Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting [27.081250446161114]
  This paper introduces the Mesh-adsorbed Gaussian Splatting (MaGS) method to address this challenge. MaGS constrains 3D Gaussians to roam near the mesh, creating a mutually adsorbed mesh-Gaussian 3D representation. Such representation harnesses both the rendering flexibility of 3D Gaussians and the structured property of meshes.
  arXiv  Detail & Related papers  (2024-06-03T17:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.