Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch

• URL: http://arxiv.org/abs/2604.09100v1
• Date: Fri, 10 Apr 2026 08:32:51 GMT
• Title: Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch
• Abstract summary: We propose a multimodal, physically grounded approach for metric-scale amodal object reconstruction and pose estimation under hand occlusion.<n>We leverage physical interaction signals: proprioception provides the posed hand geometry, and multi-contact touch constrains where the object surface must lie.
• Score: 3.926587614210279
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a multimodal, physically grounded approach for metric-scale amodal object reconstruction and pose estimation under severe hand occlusion. Unlike prior occlusion-aware 3D generation methods that rely only on vision, we leverage physical interaction signals: proprioception provides the posed hand geometry, and multi-contact touch constrains where the object surface must lie, reducing ambiguity in occluded regions. We represent object structure as a pose-aware, camera-aligned signed distance field (SDF) and learn a compact latent space with a Structure-VAE. In this latent space, we train a conditional flow-matching diffusion model, pretraining on vision-only images and finetuning on occluded manipulation scenes while conditioning on visible RGB evidence, occluder/visibility masks, the hand latent representation, and tactile information. Crucially, we incorporate physics-based objectives and differentiable decoder-guidance during finetuning and inference to reduce hand--object interpenetration and to align the reconstructed surface with contact observations. Because our method produces a metric, physically consistent structure estimate, it integrates naturally into existing two-stage reconstruction pipelines, where a downstream module refines geometry and predicts appearance. Experiments in simulation show that adding proprioception and touch substantially improves completion under occlusion and yields physically plausible reconstructions at correct real-world scale compared to vision-only baselines; we further validate transfer by deploying the model on a real humanoid robot with an end-effector different from those used during training.

Related papers

• Articulated Object Reconstruction from Rest-State Observation [33.09154459685742]
  Building interactive digital twins requires recovering both 3D geometry and the kinematic structures that govern how objects articulate.<n>We introduce a rest-state formulation that reconstructs articulated objects from a single closed configuration.
  arXiv  Detail & Related papers  (2026-07-30T06:42:03Z)
• UnfoldArt: Zero-Shot Recovery of Full Articulated 3D Objects from Text or Image [58.40343808003224]
  We present a debate-driven approach to articulated 3D object reconstruction from text or image inputs.<n>High-level agents reason about object semantics and motion using knowledge from vision-language and video models.<n>Low-level agents estimate articulation parameters and interaction points.<n>The same video prior, conditioned on the agreed articulation, then drives each part through its motion to expose occluded interiors and geometry.
  arXiv  Detail & Related papers  (2026-06-29T17:44:53Z)
• Occlusion-Robust Multi-Object Decoupling for Physics-Based Robotic Interaction [33.71202994777273]
  We propose a mask-free method for multi-object 3D reconstruction from sparse and occluded real-world views.<n>Our method produces simulation-ready 3D objects without requiring manual masks.
  arXiv  Detail & Related papers  (2026-06-28T09:50:46Z)
• TouchAnything: Diffusion-Guided 3D Reconstruction from Sparse Robot Touches [18.3533066960967]
  We present TouchAnything, a framework that leverages a pretrained vision diffusion model as a semantic and geometric prior for 3D reconstruction from sparse tactile measurements.<n>Our method reconstructs accurate geometries from only a few touches, outperforms existing baselines, and enables open-world 3D reconstruction of previously unseen object instances.
  arXiv  Detail & Related papers  (2026-04-10T04:26:36Z)
• URDF-Anything+: Autoregressive Articulated 3D Models Generation for Physical Simulation [45.4820195450296]
  We present an end-to-end autoregressive framework that directly generates executable articulated object models from visual observations.<n>Building on this capability, we enable a new Real-Follow-Sim paradigm, where high-fidelity digital twins constructed from visual observations allow policies trained in simulation to transfer to real robots without online adaptation.
  arXiv  Detail & Related papers  (2026-03-14T16:23:44Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning [11.05191602561784]
  Reconstructing physically valid 3D scenes from single-view observations is a prerequisite for bridging the gap between visual perception and robotic control.<n>We propose a novel physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data.
  arXiv  Detail & Related papers  (2026-02-13T05:24:58Z)
• Monocular Reconstruction of Neural Tactile Fields [14.002981599280787]
  We introduce neural tactile fields, a novel 3D representation that maps spatial locations to the expected tactile response upon contact.<n>Our model predicts these neural tactile fields from a single monocular RGB image -- the first method to do so.<n>When integrated with off-the-shelf path planners, neural tactile fields enable robots to generate paths that avoid high-resistance objects.
  arXiv  Detail & Related papers  (2026-02-13T01:25:19Z)
• ArtiLatent: Realistic Articulated 3D Object Generation via Structured Latents [31.495577251319315]
  ArtiLatent is a generative framework that synthesizes human-made 3D objects with fine-grained geometry, accurate articulation, and realistic appearance.
  arXiv  Detail & Related papers  (2025-10-24T13:08:15Z)
• Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance [61.41904916189093]
  We propose a novel diffusion-based framework for reconstructing 3D geometry of hand-held objects from monocular RGB images.<n>We use hand-object interaction as geometric guidance to ensure plausible hand-object interactions.
  arXiv  Detail & Related papers  (2025-08-25T17:11:53Z)
• Decaf: Monocular Deformation Capture for Face and Hand Interactions [77.75726740605748]
  This paper introduces the first method that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system.
  arXiv  Detail & Related papers  (2023-09-28T17:59:51Z)
• Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery [70.66865453410958]
  Articulation-centric 2D/3D pose supervision forms the core training objective in most existing 3D human pose estimation techniques. We propose a novel framework that relies only on silhouette supervision to adapt a source-trained model-based regressor. We develop a series of convolution-friendly spatial transformations in order to disentangle a topological-skeleton representation from the raw silhouette.
  arXiv  Detail & Related papers  (2022-04-04T06:58:15Z)
• Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments [81.38641691636847]
  We rethink the problem of scene reconstruction from an embodied agent's perspective. We reconstruct an interactive scene using RGB-D data stream. This reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models.
  arXiv  Detail & Related papers  (2021-03-30T05:56:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.