Tactile MNIST: Benchmarking Active Tactile Perception

• URL: http://arxiv.org/abs/2506.06361v2
• Date: Sat, 14 Jun 2025 14:33:43 GMT
• Title: Tactile MNIST: Benchmarking Active Tactile Perception
• Abstract summary: We introduce the Tactile MNIST Benchmark Suite, an open-source, Gymnasium-compatible benchmark for tactile perception tasks.<n>Our benchmark suite offers diverse simulation scenarios, from simple toy environments all the way to complex tactile perception tasks using vision-based tactile sensors.<n>We also offer a comprehensive dataset comprising 13,500 synthetic 3D MNIST digit models and 153,600 real-world tactile samples collected from 600 3D printed digits.
• Score: 19.93022179513013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tactile perception has the potential to significantly enhance dexterous robotic manipulation by providing rich local information that can complement or substitute for other sensory modalities such as vision. However, because tactile sensing is inherently local, it is not well-suited for tasks that require broad spatial awareness or global scene understanding on its own. A human-inspired strategy to address this issue is to consider active perception techniques instead. That is, to actively guide sensors toward regions with more informative or significant features and integrate such information over time in order to understand a scene or complete a task. Both active perception and different methods for tactile sensing have received significant attention recently. Yet, despite advancements, both fields lack standardized benchmarks. To bridge this gap, we introduce the Tactile MNIST Benchmark Suite, an open-source, Gymnasium-compatible benchmark specifically designed for active tactile perception tasks, including localization, classification, and volume estimation. Our benchmark suite offers diverse simulation scenarios, from simple toy environments all the way to complex tactile perception tasks using vision-based tactile sensors. Furthermore, we also offer a comprehensive dataset comprising 13,500 synthetic 3D MNIST digit models and 153,600 real-world tactile samples collected from 600 3D printed digits. Using this dataset, we train a CycleGAN for realistic tactile simulation rendering. By providing standardized protocols and reproducible evaluation frameworks, our benchmark suite facilitates systematic progress in the fields of tactile sensing and active perception.

Related papers

• $\mathcal{N}_0$-Foundation: Towards the Age of Tactile Intelligence [0.0]
  $mathcalN$-Foundation is a paradigm for tactile-enabled embodied manipulation.<n>We engineer the infrastructure for scalable data collection, including a vision-based tactile sensor, a tactile Universal Manipulation Interface (UMI), and a synchronized visuo-tactile data collection system.<n>We construct NeoData, which contains more than 30000 hours of synchronized visual and tactile demonstrations, spanning six embodiments, 450 tasks, and billions of paired RGB and tactile frames collected through a mixture of real-robot teleoperation and UMI-based demonstrations.
  arXiv  Detail & Related papers  (2026-08-30T06:46:18Z)
• Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor [52.33204893988409]
  We propose a visuo-tactile sensor featuring customized uniform illumination, alongside a unified perception framework.<n>The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
  arXiv  Detail & Related papers  (2026-08-25T07:25:27Z)
• HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision [7.104046531938793]
  HT-Bench is a large-scale benchmark for dexterous full-hand tactile sensing.<n>HandTouch is a vector-quantized vision--tactile encoder that learns tactile representations through progressive spatial, cross-modal, and temporal training.
  arXiv  Detail & Related papers  (2026-06-17T15:01:30Z)
• Tac-DINO: Learning Vision-Tactile Features with Patch Alignment [47.4807111947232]
  We build a data collection system to acquire a large-scale tactile dataset, with over 20 K tactile contacts from 505 real-world objects.<n>We then design a Vis-Tac Holographic Matching Benchmark to evaluate vision-tactile local-to-global alignment ability.<n>Then we propose Vision-Tactile Patch Alignment (VTPA) methods for vision-tactile representation learning.
  arXiv  Detail & Related papers  (2026-06-10T13:33:42Z)
• TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation [50.608989079323784]
  We propose TouchThinker, a tactile-language framework that scales tactile commonsense reasoning to the open world from both data and representation perspectives.<n>First, we construct TouchThinker-1M, a million-scale, multi-source tactile reasoning dataset covering textbf415 objects, textbf8 scenarios, and textbf7 sensor types.<n>Then, we propose action-aware modeling mechanism to improve tactile representation efficiency and enable efficient reasoning.
  arXiv  Detail & Related papers  (2026-06-10T03:58:32Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking [47.900334665380115]
  We propose UniVTAC, a simulation-based visuo-tactile data platform.<n>We introduce the UniVTAC encoder, a visuo-tactile encoder trained on large-scale simulation-synthesized data.<n>We present the UniVTAC Benchmark, which consists of eight representative visuo-tactile manipulation tasks.
  arXiv  Detail & Related papers  (2026-02-10T18:57:00Z)
• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction [93.88239833545623]
  We present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset.<n>We show that tactile signals provide a compact yet powerful cue for grasp understanding.<n>We aim to advance multimodal egocentric perception, embodied learning, and contact-rich robotic manipulation.
  arXiv  Detail & Related papers  (2025-12-18T18:18:17Z)
• SensorLM: Learning the Language of Wearable Sensors [50.95988682423808]
  We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language.<n>We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data.<n>This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people.
  arXiv  Detail & Related papers  (2025-06-10T17:13:09Z)
• Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Setting [25.355424080824996]
  Tactile sensing is critical for human perception and underpins applications in computer vision, robotics, and multimodal learning.<n>Because tactile data is often scarce and costly to acquire, generating synthetic tactile images provides a scalable solution to augment real-world measurements.<n>We demonstrate that overlapping training and test samples in commonly used datasets inflate performance metrics, obscuring the true generalizability of tactile models.
  arXiv  Detail & Related papers  (2025-03-10T02:37:22Z)
• AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors [11.506370451126378]
  Visuo-tactile sensors aim to emulate human tactile perception, enabling robots to understand and manipulate objects.<n>We introduce TacQuad, an aligned multi-modal tactile multi-sensor dataset from four different visuo-tactile sensors.<n>We propose AnyTouch, a unified static-dynamic multi-sensor representation learning framework with a multi-level structure.
  arXiv  Detail & Related papers  (2025-02-15T08:33:25Z)
• Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control [12.302685367517718]
  High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. We study a new concept: textittactile saliency for robot touch, inspired by the human touch attention mechanism from neuroscience.
  arXiv  Detail & Related papers  (2023-07-26T21:19:45Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• AGO-Net: Association-Guided 3D Point Cloud Object Detection Network [86.10213302724085]
  We propose a novel 3D detection framework that associates intact features for objects via domain adaptation. We achieve new state-of-the-art performance on the KITTI 3D detection benchmark in both accuracy and speed.
  arXiv  Detail & Related papers  (2022-08-24T16:54:38Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• Active 3D Shape Reconstruction from Vision and Touch [66.08432412497443]
  Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. In 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings. We introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile priors to guide the shape exploration; and 3) a set of data-driven solutions with either tactile or visuo
  arXiv  Detail & Related papers  (2021-07-20T15:56:52Z)
• Spatio-temporal Attention Model for Tactile Texture Recognition [25.06942319117782]
  We propose a novel Spatio-Temporal Attention Model (STAM) for tactile texture recognition. The proposed STAM pays attention to both spatial focus of each single tactile texture and the temporal correlation of a tactile sequence. In the experiments to discriminate 100 different fabric textures, the spatially and temporally selective attention has resulted in a significant improvement of the recognition accuracy.
  arXiv  Detail & Related papers  (2020-08-10T22:32:34Z)
• Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing [1.9981375888949469]
  This paper focuses on a vision-based tactile sensor, which aims to reconstruct the distribution of the three-dimensional contact forces applied on its soft surface. A strategy is proposed to train a tailored deep neural network entirely from the simulation data. The resulting learning architecture is directly transferable across multiple tactile sensors without further training and yields accurate predictions on real data.
  arXiv  Detail & Related papers  (2020-03-05T14:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.