Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces

• URL: http://arxiv.org/abs/2403.20275v1
• Date: Fri, 29 Mar 2024 16:30:17 GMT
• Title: Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces
• Authors: Mauro Comi, Alessio Tonioni, Max Yang, Jonathan Tremblay, Valts Blukis, Yijiong Lin, Nathan F. Lepora, Laurence Aitchison,
• Abstract summary: We propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view synthesis. By creating a framework that decreases the transmittance at touch locations, we achieve a refined surface reconstruction, ensuring a uniformly smooth depth map. We conduct evaluation on objects with glossy and reflective surfaces and demonstrate the effectiveness of our approach.
• Score: 34.831730064258494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Touch and vision go hand in hand, mutually enhancing our ability to understand the world. From a research perspective, the problem of mixing touch and vision is underexplored and presents interesting challenges. To this end, we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view synthesis. Our method optimises 3D Gaussian primitives to accurately model the object's geometry at points of contact. By creating a framework that decreases the transmittance at touch locations, we achieve a refined surface reconstruction, ensuring a uniformly smooth depth map. Touch is particularly useful when considering non-Lambertian objects (e.g. shiny or reflective surfaces) since contemporary methods tend to fail to reconstruct with fidelity specular highlights. By combining vision and tactile sensing, we achieve more accurate geometry reconstructions with fewer images than prior methods. We conduct evaluation on objects with glossy and reflective surfaces and demonstrate the effectiveness of our approach, offering significant improvements in reconstruction quality.

Related papers

• GlossyGS: Inverse Rendering of Glossy Objects with 3D Gaussian Splatting [21.23724172779984]
  GlossyGS aims to precisely reconstruct the geometry and materials of glossy objects by integrating material priors. We demonstrate through quantitative analysis and qualitative visualization that the proposed method is effective to reconstruct high-fidelity geometries and materials of glossy objects.
  arXiv  Detail & Related papers  (2024-10-17T09:00:29Z)
• MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References [49.71130133080821]
  MaRINeR is a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint. We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations.
  arXiv  Detail & Related papers  (2024-07-18T17:50:03Z)
• Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images [59.49985837246644]
  We show how to model contacts in an explicit way to benefit the implicit reconstruction of hand-held objects. In the first part, we propose a new subtask of directly estimating 3D hand-object contacts from a single image. In the second part, we introduce a novel method to diffuse estimated contact states from the hand mesh surface to nearby 3D space.
  arXiv  Detail & Related papers  (2023-05-31T17:59:26Z)
• Recovering Fine Details for Neural Implicit Surface Reconstruction [3.9702081347126943]
  We present D-NeuS, a volume rendering neural implicit surface reconstruction method capable to recover fine geometry details. We impose multi-view feature consistency on the surface points, derived by interpolating SDF zero-crossings from sampled points along rays. Our method reconstructs high-accuracy surfaces with details, and outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-11-21T10:06:09Z)
• Active 3D Shape Reconstruction from Vision and Touch [66.08432412497443]
  Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. In 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings. We introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile priors to guide the shape exploration; and 3) a set of data-driven solutions with either tactile or visuo
  arXiv  Detail & Related papers  (2021-07-20T15:56:52Z)
• Multi-view 3D Reconstruction of a Texture-less Smooth Surface of Unknown Generic Reflectance [86.05191217004415]
  Multi-view reconstruction of texture-less objects with unknown surface reflectance is a challenging task. This paper proposes a simple and robust solution to this problem based on a co-light scanner.
  arXiv  Detail & Related papers  (2021-05-25T01:28:54Z)
• Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images [64.53227129573293]
  We investigate the problem of learning to generate 3D parametric surface representations for novel object instances, as seen from one or more views. We design neural networks capable of generating high-quality parametric 3D surfaces which are consistent between views. Our method is supervised and trained on a public dataset of shapes from common object categories.
  arXiv  Detail & Related papers  (2020-08-18T06:33:40Z)
• 3D Shape Reconstruction from Vision and Touch [62.59044232597045]
  In 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. We introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects.
  arXiv  Detail & Related papers  (2020-07-07T20:20:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.