An Interpretable Deep Learning Framework for Material Perception and Classification from Multisensory Tactile Data

• URL: http://arxiv.org/abs/2608.21894v1
• Date: Sat, 22 Aug 2026 10:13:46 GMT
• Title: An Interpretable Deep Learning Framework for Material Perception and Classification from Multisensory Tactile Data
• Abstract summary: We develop a framework for mapping multisensory touch data to material perception.<n>We show that deep learning can approach near-perfect material classification when unconstrained by intermediate perceptual stages.<n>We also show how interpretable deep learning can both approach human-level performance and reveal cues that robotic and haptic systems need to incorporate.
• Score: 3.7277730514654555
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human tactile perception relies on complex multisensory cues. Yet the relationship between tactile signals and perceptual representations remains poorly understood, limiting the integration of touch in digital environments and human-like robotic perception. To address this gap, we developed a computational framework comprising three interconnected deep learning models that map multisensory touch data to material perception, without relying on hand-crafted features. The models represent progressively different routes from tactile signals to material class: from low-level interaction signals to perceptual attribute distributions (Model 1), from predicted attribute distributions to material classification (Model 2), and directly from tactile signals to material categories, bypassing intermediate representations (Model 3). By combining deep learning with Integrated Gradients, the framework achieved high accuracy while offering interpretability, revealing which sensory modalities most strongly drive its decisions. Our results show that deep learning can approach near-perfect material classification when unconstrained by intermediate perceptual stages, but matching human-like performance is harder once those stages are modeled explicitly. Notably, thermal cues emerged as particularly informative across all models, providing robust signals for material differentiation. The results offer a computational account of how tactile signals lead to material perception and show how interpretable deep learning can both approach human-level performance and reveal cues that robotic and haptic systems need to incorporate.

Related papers

• AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion [60.79507611737292]
  Humans achieve stable and adaptive grasps by seamlessly integrating visual perception and tactile feedback.<n>Existing robotic grasping approaches rely on visual inputs and lack mechanisms for tactile-guided adaptation after contact.<n>We propose a unified visuo-tactile-fusion grasping framework that integrates grasp generation, feasibility prediction, and adaptive refinement.
  arXiv  Detail & Related papers  (2026-08-06T16:29:11Z)
• Learning to Feel Materials from Multisensory Tactile Data via Interpretable Models [4.189643331553923]
  We present an interpretable computational framework for modeling human material perception and recognition using multisensory touch data.<n>Results show that combining information from pressing, static contact, and sliding interactions improves prediction accuracy.<n>These findings highlight the importance of thermal and compliance cues, which remain underrepresented in current robotic fingers and haptic displays.
  arXiv  Detail & Related papers  (2026-05-28T08:20:01Z)
• Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training [0.850206009406913]
  MViTac is a novel methodology that leverages contrastive learning to integrate vision and touch sensations in a self-supervised fashion. By availing both sensory inputs, MViTac leverages intra and inter-modality losses for learning representations, resulting in enhanced material property classification and more adept grasping prediction.
  arXiv  Detail & Related papers  (2024-01-22T15:11:57Z)
• Bayesian and Neural Inference on LSTM-based Object Recognition from Tactile and Kinesthetic Information [0.0]
  Haptic perception encompasses the sensing modalities encountered in the sense of touch (e.g., tactile and kinesthetic sensations) This letter focuses on multimodal object recognition and proposes analytical and data-driven methodologies to fuse tactile- and kinesthetic-based classification results.
  arXiv  Detail & Related papers  (2023-06-10T12:29:23Z)
• Neurosymbolic hybrid approach to driver collision warning [64.02492460600905]
  There are two main algorithmic approaches to autonomous driving systems. Deep learning alone has achieved state-of-the-art results in many areas. But sometimes it can be very difficult to debug if the deep learning model doesn't work.
  arXiv  Detail & Related papers  (2022-03-28T20:29:50Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)
• Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules [81.1967157385085]
  Robust perception relies on both bottom-up and top-down signals. We explore deep recurrent neural net architectures in which bottom-up and top-down signals are dynamically combined using attention. We demonstrate on a variety of benchmarks in language modeling, sequential image classification, video prediction and reinforcement learning that the emphbidirectional information flow can improve results over strong baselines.
  arXiv  Detail & Related papers  (2020-06-30T17:26:19Z)
• A Framework for Learning Invariant Physical Relations in Multimodal Sensory Processing [0.0]
  We design a novel neural network architecture capable of learning, in an unsupervised manner, relations among sensory cues. We describe the core system functionality when learning arbitrary non-linear relations in low-dimensional sensory data. We demonstrate this through a real-world learning problem, where, from standard RGB camera frames, the network learns the relations between physical quantities.
  arXiv  Detail & Related papers  (2020-06-30T08:42:48Z)
• Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing [1.9981375888949469]
  This paper focuses on a vision-based tactile sensor, which aims to reconstruct the distribution of the three-dimensional contact forces applied on its soft surface. A strategy is proposed to train a tailored deep neural network entirely from the simulation data. The resulting learning architecture is directly transferable across multiple tactile sensors without further training and yields accurate predictions on real data.
  arXiv  Detail & Related papers  (2020-03-05T14:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.