Adding internal audio sensing to internal vision enables human-like in-hand fabric recognition with soft robotic fingertips

• URL: http://arxiv.org/abs/2602.12918v1
• Date: Fri, 13 Feb 2026 13:23:52 GMT
• Title: Adding internal audio sensing to internal vision enables human-like in-hand fabric recognition with soft robotic fingertips
• Abstract summary: In this work, we present a system that can sense both types of haptic information.<n>Inspired by movements humans make to evaluate fabrics, our robot actively encloses and rubs fabric samples between its two sensitive fingers.
• Score: 24.17784887837483
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Distinguishing the feel of smooth silk from coarse cotton is a trivial everyday task for humans. When exploring such fabrics, fingertip skin senses both spatio-temporal force patterns and texture-induced vibrations that are integrated to form a haptic representation of the explored material. It is challenging to reproduce this rich, dynamic perceptual capability in robots because tactile sensors typically cannot achieve both high spatial resolution and high temporal sampling rate. In this work, we present a system that can sense both types of haptic information, and we investigate how each type influences robotic tactile perception of fabrics. Our robotic hand's middle finger and thumb each feature a soft tactile sensor: one is the open-source Minsight sensor that uses an internal camera to measure fingertip deformation and force at 50 Hz, and the other is our new sensor Minsound that captures vibrations through an internal MEMS microphone with a bandwidth from 50 Hz to 15 kHz. Inspired by the movements humans make to evaluate fabrics, our robot actively encloses and rubs folded fabric samples between its two sensitive fingers. Our results test the influence of each sensing modality on overall classification performance, showing high utility for the audio-based sensor. Our transformer-based method achieves a maximum fabric classification accuracy of 97 % on a dataset of 20 common fabrics. Incorporating an external microphone away from Minsound increases our method's robustness in loud ambient noise conditions. To show that this audio-visual tactile sensing approach generalizes beyond the training data, we learn general representations of fabric stretchiness, thickness, and roughness.

Related papers

• Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor [52.33204893988409]
  We propose a visuo-tactile sensor featuring customized uniform illumination, alongside a unified perception framework.<n>The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
  arXiv  Detail & Related papers  (2026-08-25T07:25:27Z)
• Acoustic Sensing for Universal Jamming Grippers [42.48594146171026]
  Sound propagates through the gripper and object, encoding object properties, which are then reconstructed via machine learning.<n>Our sensor achieves high spatial resolution in sensing object size (2.6 mm error) and orientation (0.6 deg error)<n>We validate the sensor in a realistic tactile object sorting task, achieving 53 minutes of uninterrupted grasping and sensing.
  arXiv  Detail & Related papers  (2026-02-27T22:31:05Z)
• Vibro-Sense: Robust Vibration-based Impulse Response Localization and Trajectory Tracking for Robotic Hands [0.5500249707065662]
  Rich contact perception is crucial for robotic manipulation, yet traditional tactile skins remain expensive and complex to integrate.<n>This paper presents a scalable alternative: high-accuracy whole-body touch localization via vibro-acoustic sensing.
  arXiv  Detail & Related papers  (2026-01-28T12:49:39Z)
• Audio-Based Tactile Human-Robot Interaction Recognition [0.0]
  This study explores the use of microphones placed on a robot's body to detect tactile interactions via sounds produced when the hard shell of the robot is touched.<n>Two I2S microphones integrated with a Raspberry Pi 4 were positioned on the torso of a Pollen Robotics Reachy robot to capture audio signals.<n>A convolutional neural network was trained for touch classification on a dataset of 336 pre-processed samples.<n>The model shows high classification accuracy between touch types with distinct acoustic dominant frequencies.
  arXiv  Detail & Related papers  (2025-12-06T23:19:22Z)
• Tactile MNIST: Benchmarking Active Tactile Perception [19.93022179513013]
  We introduce the Tactile MNIST Benchmark Suite, an open-source, Gymnasium-compatible benchmark for tactile perception tasks.<n>Our benchmark suite offers diverse simulation scenarios, from simple toy environments all the way to complex tactile perception tasks using vision-based tactile sensors.<n>We also offer a comprehensive dataset comprising 13,500 synthetic 3D MNIST digit models and 153,600 real-world tactile samples collected from 600 3D printed digits.
  arXiv  Detail & Related papers  (2025-06-03T14:42:16Z)
• Digitizing Touch with an Artificial Multimodal Fingertip [51.7029315337739]
  Humans and robots both benefit from using touch to perceive and interact with the surrounding environment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities.
  arXiv  Detail & Related papers  (2024-11-04T18:38:50Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation [49.925499720323806]
  We study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor.
  arXiv  Detail & Related papers  (2022-12-07T18:55:53Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• Extended Tactile Perception: Vibration Sensing through Tools and Grasped Objects [11.60680126397386]
  We show how we can enable robots to embody tools and extend perception using standard grasped objects. We propose that vibro-tactile sensing using dynamic tactile sensors on the robot fingers, along with machine learning models, enables robots to decipher contact information. This paper reports on extensive experiments using the BioTac micro-vibration sensor and a new event dynamic sensor, the NUSkin, capable of multi-taxel sensing at 4kHz.
  arXiv  Detail & Related papers  (2021-06-01T13:49:31Z)
• Spatio-temporal Attention Model for Tactile Texture Recognition [25.06942319117782]
  We propose a novel Spatio-Temporal Attention Model (STAM) for tactile texture recognition. The proposed STAM pays attention to both spatial focus of each single tactile texture and the temporal correlation of a tactile sequence. In the experiments to discriminate 100 different fabric textures, the spatially and temporally selective attention has resulted in a significant improvement of the recognition accuracy.
  arXiv  Detail & Related papers  (2020-08-10T22:32:34Z)
• OmniTact: A Multi-Directional High Resolution Touch Sensor [109.28703530853542]
  Existing tactile sensors are either flat, have small sensitive fields or only provide low-resolution signals. We introduce OmniTact, a multi-directional high-resolution tactile sensor. We evaluate the capabilities of OmniTact on a challenging robotic control task.
  arXiv  Detail & Related papers  (2020-03-16T01:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.