Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces From Images

• URL: http://arxiv.org/abs/2004.14487v3
• Date: Mon, 20 Sep 2021 19:03:56 GMT
• Title: Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces From Images
• Authors: Matthew Purri and Kristin Dana
• Abstract summary: We introduce the challenging task of estimating a set of tactile physical properties from visual information. We construct a first of its kind image-tactile dataset with over 400 multiview image sequences and the corresponding tactile properties. We develop a cross-modal framework comprised of an adversarial objective and a novel visuo-tactile joint classification loss.
• Score: 4.666400601228301
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The connection between visual input and tactile sensing is critical for object manipulation tasks such as grasping and pushing. In this work, we introduce the challenging task of estimating a set of tactile physical properties from visual information. We aim to build a model that learns the complex mapping between visual information and tactile physical properties. We construct a first of its kind image-tactile dataset with over 400 multiview image sequences and the corresponding tactile properties. A total of fifteen tactile physical properties across categories including friction, compliance, adhesion, texture, and thermal conductance are measured and then estimated by our models. We develop a cross-modal framework comprised of an adversarial objective and a novel visuo-tactile joint classification loss. Additionally, we develop a neural architecture search framework capable of selecting optimal combinations of viewing angles for estimating a given physical property.

Related papers

• Compositional Physical Reasoning of Objects and Events from Videos [122.6862357340911]
  This paper addresses the challenge of inferring hidden physical properties from objects' motion and interactions. We evaluate state-of-the-art video reasoning models on ComPhy and reveal their limited ability to capture these hidden properties. We also propose a novel neuro-symbolic framework, Physical Concept Reasoner (PCR), that learns and reasons about both visible and hidden physical properties.
  arXiv  Detail & Related papers  (2024-08-02T15:19:55Z)
• Controllable Visual-Tactile Synthesis [28.03469909285511]
  We develop a conditional generative model that synthesizes both visual and tactile outputs from a single sketch. We then introduce a pipeline to render high-quality visual and tactile outputs on an electroadhesion-based haptic device.
  arXiv  Detail & Related papers  (2023-05-04T17:59:51Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• Touch and Go: Learning from Human-Collected Vision and Touch [16.139106833276]
  We propose a dataset with paired visual and tactile data called Touch and Go. Human data collectors probe objects in natural environments using tactile sensors. Our dataset spans a large number of "in the wild" objects and scenes.
  arXiv  Detail & Related papers  (2022-11-22T18:59:32Z)
• VisTaNet: Attention Guided Deep Fusion for Surface Roughness Classification [0.0]
  This paper presents a visual dataset that augments an existing tactile dataset. We propose a novel deep fusion architecture that fuses visual and tactile data using four types of fusion strategies. Our model shows significant performance improvements (97.22%) in surface roughness classification accuracy over tactile only.
  arXiv  Detail & Related papers  (2022-09-18T09:37:06Z)
• ComPhy: Compositional Physical Reasoning of Objects and Events from Videos [113.2646904729092]
  The compositionality between the visible and hidden properties poses unique challenges for AI models to reason from the physical world. Existing studies on video reasoning mainly focus on visually observable elements such as object appearance, movement, and contact interaction. We propose an oracle neural-symbolic framework named Compositional Physics Learner (CPL), combining visual perception, physical property learning, dynamic prediction, and symbolic execution.
  arXiv  Detail & Related papers  (2022-05-02T17:59:13Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• Learning Intuitive Physics with Multimodal Generative Models [24.342994226226786]
  This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes. We use a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces. We validate through simulated and real-world experiments in which the resting state of an object is predicted from given initial conditions.
  arXiv  Detail & Related papers  (2021-01-12T12:55:53Z)
• Visual Grounding of Learned Physical Models [66.04898704928517]
  Humans intuitively recognize objects' physical properties and predict their motion, even when the objects are engaged in complicated interactions. We present a neural model that simultaneously reasons about physics and makes future predictions based on visual and dynamics priors. Experiments show that our model can infer the physical properties within a few observations, which allows the model to quickly adapt to unseen scenarios and make accurate predictions into the future.
  arXiv  Detail & Related papers  (2020-04-28T17:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.