Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction

• URL: http://arxiv.org/abs/2505.13091v1
• Date: Mon, 19 May 2025 13:24:21 GMT
• Title: Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction
• Authors: Yuanbo Wang, Zhaoxuan Zhang, Jiajin Qiu, Dilong Sun, Zhengyu Meng, Xiaopeng Wei, Xin Yang,
• Abstract summary: Current 3D diffusion models focus on reconstructing target shape from images or a set of partial observations.<n>We propose a Touch2Shape model, which leverages a touch-conditioned diffusion model to explore and reconstruct the target shape from touch.<n>For shape reconstruction, we have developed a touch embedding module to condition the diffusion model in creating a compact representation.<n>For shape exploration, we combine the diffusion model with reinforcement learning to train a policy.
• Score: 7.815446497747986
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have made breakthroughs in 3D generation tasks. Current 3D diffusion models focus on reconstructing target shape from images or a set of partial observations. While excelling in global context understanding, they struggle to capture the local details of complex shapes and limited to the occlusion and lighting conditions. To overcome these limitations, we utilize tactile images to capture the local 3D information and propose a Touch2Shape model, which leverages a touch-conditioned diffusion model to explore and reconstruct the target shape from touch. For shape reconstruction, we have developed a touch embedding module to condition the diffusion model in creating a compact representation and a touch shape fusion module to refine the reconstructed shape. For shape exploration, we combine the diffusion model with reinforcement learning to train a policy. This involves using the generated latent vector from the diffusion model to guide the touch exploration policy training through a novel reward design. Experiments validate the reconstruction quality thorough both qualitatively and quantitative analysis, and our touch exploration policy further boosts reconstruction performance.

Related papers

• HORT: Monocular Hand-held Objects Reconstruction with Transformers [61.36376511119355]
  Reconstructing hand-held objects in 3D from monocular images is a significant challenge in computer vision.<n>We propose a transformer-based model to efficiently reconstruct dense 3D point clouds of hand-held objects.<n>Our method achieves state-of-the-art accuracy with much faster inference speed, while generalizing well to in-the-wild images.
  arXiv  Detail & Related papers  (2025-03-27T09:45:09Z)
• Hyperbolic-constraint Point Cloud Reconstruction from Single RGB-D Images [19.23499128175523]
  We introduce hyperbolic space to 3D point cloud reconstruction, enabling the model to represent and understand complex hierarchical structures in point clouds with low distortion.<n>Our model outperforms most existing models, and ablation studies demonstrate the significance of our model and its components.
  arXiv  Detail & Related papers  (2024-12-12T08:27:39Z)
• Deformable 3D Shape Diffusion Model [21.42513407755273]
  We introduce a novel deformable 3D shape diffusion model that facilitates comprehensive 3D shape manipulation. We demonstrate state-of-the-art performance in point cloud generation and competitive results in mesh deformation. Our method presents a unique pathway for advancing 3D shape manipulation and unlocking new opportunities in the realm of virtual reality.
  arXiv  Detail & Related papers  (2024-07-31T08:24:42Z)
• SC-Diff: 3D Shape Completion with Latent Diffusion Models [4.913210912019975]
  This paper introduces a 3D shape completion approach using a 3D latent diffusion model optimized for completing shapes. Our method combines image-based conditioning through cross-attention and spatial conditioning through the integration of 3D features from captured partial scans.
  arXiv  Detail & Related papers  (2024-03-19T06:01:11Z)
• NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion [56.98287481620215]
  We present a novel method for 3D surface reconstruction from multiple images where only a part of the object of interest is captured. Our approach builds on two recent developments: surface reconstruction using neural radiance fields for the reconstruction of the visible parts of the surface, and guidance of pre-trained 2D diffusion models in the form of Score Distillation Sampling (SDS) to complete the shape in unobserved regions in a plausible manner.
  arXiv  Detail & Related papers  (2023-12-07T19:30:55Z)
• Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models [83.35835521670955]
  Surf-D is a novel method for generating high-quality 3D shapes as Surfaces with arbitrary topologies. We use the Unsigned Distance Field (UDF) as our surface representation to accommodate arbitrary topologies. We also propose a new pipeline that employs a point-based AutoEncoder to learn a compact and continuous latent space for accurately encoding UDF.
  arXiv  Detail & Related papers  (2023-11-28T18:56:01Z)
• SADIR: Shape-Aware Diffusion Models for 3D Image Reconstruction [2.2954246824369218]
  3D image reconstruction from a limited number of 2D images has been a long-standing challenge in computer vision and image analysis. We propose a shape-aware network based on diffusion models for 3D image reconstruction, named SADIR, to address these issues.
  arXiv  Detail & Related papers  (2023-09-06T19:30:22Z)
• Generative Deformable Radiance Fields for Disentangled Image Synthesis of Topology-Varying Objects [52.46838926521572]
  3D-aware generative models have demonstrated their superb performance to generate 3D neural radiance fields (NeRF) from a collection of monocular 2D images. We propose a generative model for synthesizing radiance fields of topology-varying objects with disentangled shape and appearance variations.
  arXiv  Detail & Related papers  (2022-09-09T08:44:06Z)
• SP-GAN: Sphere-Guided 3D Shape Generation and Manipulation [50.53931728235875]
  We present SP-GAN, a new unsupervised sphere-guided generative model for direct synthesis of 3D shapes in the form of point clouds. Compared with existing models, SP-GAN is able to synthesize diverse and high-quality shapes with fine details.
  arXiv  Detail & Related papers  (2021-08-10T06:49:45Z)
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth [64.29043589521308]
  We propose a rendering module to augment the training data by synthesizing images with virtual-depths. The rendering module takes as input the RGB image and its corresponding sparse depth image, outputs a variety of photo-realistic synthetic images. Besides, we introduce an auxiliary module to improve the detection model by jointly optimizing it through a depth estimation task.
  arXiv  Detail & Related papers  (2021-07-28T11:00:47Z)
• Active 3D Shape Reconstruction from Vision and Touch [66.08432412497443]
  Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. In 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings. We introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile priors to guide the shape exploration; and 3) a set of data-driven solutions with either tactile or visuo
  arXiv  Detail & Related papers  (2021-07-20T15:56:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.