Self-supervised Spatio-Temporal Graph Mask-Passing Attention Network for Perceptual Importance Prediction of Multi-point Tactility

• URL: http://arxiv.org/abs/2410.03434v1
• Date: Fri, 4 Oct 2024 13:45:50 GMT
• Title: Self-supervised Spatio-Temporal Graph Mask-Passing Attention Network for Perceptual Importance Prediction of Multi-point Tactility
• Authors: Dazhong He, Qian Liu,
• Abstract summary: We develop a model to predict tactile perceptual importance at multiple points, based on self-supervised learning and Spatio-Temporal Graph Neural Network. Results indicate that this model can effectively predict the perceptual importance of various points in multi-point tactile perception scenarios.
• Score: 8.077951761948556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While visual and auditory information are prevalent in modern multimedia systems, haptic interaction, e.g., tactile and kinesthetic interaction, provides a unique form of human perception. However, multimedia technology for contact interaction is less mature than non-contact multimedia technologies and requires further development. Specialized haptic media technologies, requiring low latency and bitrates, are essential to enable haptic interaction, necessitating haptic information compression. Existing vibrotactile signal compression methods, based on the perceptual model, do not consider the characteristics of fused tactile perception at multiple spatially distributed interaction points. In fact, differences in tactile perceptual importance are not limited to conventional frequency and time domains, but also encompass differences in the spatial locations on the skin unique to tactile perception. For the most frequently used tactile information, vibrotactile texture perception, we have developed a model to predict its perceptual importance at multiple points, based on self-supervised learning and Spatio-Temporal Graph Neural Network. Current experimental results indicate that this model can effectively predict the perceptual importance of various points in multi-point tactile perception scenarios.

Related papers

• Multimodal Adaptive Fusion of Face and Gait Features using Keyless attention based Deep Neural Networks for Human Identification [67.64124512185087]
  Soft biometrics such as gait are widely used with face in surveillance tasks like person recognition and re-identification. We propose a novel adaptive multi-biometric fusion strategy for the dynamic incorporation of gait and face biometric cues by leveraging keyless attention deep neural networks.
  arXiv  Detail & Related papers  (2023-03-24T05:28:35Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• Multimodal perception for dexterous manipulation [14.314776558032166]
  We propose a cross-modal sensory data generation framework for the translation between vision and touch. We propose a-temporal attention model for tactile texture recognition, which takes both spatial features and time dimension into consideration.
  arXiv  Detail & Related papers  (2021-12-28T21:20:26Z)
• A Framework for Multisensory Foresight for Embodied Agents [11.351546861334292]
  Predicting future sensory states is crucial for learning agents such as robots, drones, and autonomous vehicles. In this paper, we couple multiple sensory modalities with exploratory actions and propose a predictive neural network architecture to address this problem. The framework was tested and validated with a dataset containing 4 sensory modalities (vision, haptic, audio, and tactile) on a humanoid robot performing 9 behaviors multiple times on a large set of objects.
  arXiv  Detail & Related papers  (2021-09-15T20:20:04Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• Learning Contact Dynamics using Physically Structured Neural Networks [81.73947303886753]
  We use connections between deep neural networks and differential equations to design a family of deep network architectures for representing contact dynamics between objects. We show that these networks can learn discontinuous contact events in a data-efficient manner from noisy observations. Our results indicate that an idealised form of touch feedback is a key component of making this learning problem tractable.
  arXiv  Detail & Related papers  (2021-02-22T17:33:51Z)
• Learning Intuitive Physics with Multimodal Generative Models [24.342994226226786]
  This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes. We use a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces. We validate through simulated and real-world experiments in which the resting state of an object is predicted from given initial conditions.
  arXiv  Detail & Related papers  (2021-01-12T12:55:53Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)
• TactileSGNet: A Spiking Graph Neural Network for Event-based Tactile Object Recognition [17.37142241982902]
  New advances in flexible, event-driven, electronic skins may soon endow robots with touch perception capabilities similar to humans. These unique features may render current deep learning approaches such as convolutional feature extractors unsuitable for tactile learning. We propose a novel spiking graph neural network for event-based tactile object recognition.
  arXiv  Detail & Related papers  (2020-08-01T03:35:15Z)
• A Framework for Learning Invariant Physical Relations in Multimodal Sensory Processing [0.0]
  We design a novel neural network architecture capable of learning, in an unsupervised manner, relations among sensory cues. We describe the core system functionality when learning arbitrary non-linear relations in low-dimensional sensory data. We demonstrate this through a real-world learning problem, where, from standard RGB camera frames, the network learns the relations between physical quantities.
  arXiv  Detail & Related papers  (2020-06-30T08:42:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.