HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision

• URL: http://arxiv.org/abs/2606.19161v1
• Date: Wed, 17 Jun 2026 15:01:30 GMT
• Title: HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision
• Abstract summary: HT-Bench is a large-scale benchmark for dexterous full-hand tactile sensing.<n>HandTouch is a vector-quantized vision--tactile encoder that learns tactile representations through progressive spatial, cross-modal, and temporal training.
• Score: 7.104046531938793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Establishing a universal benchmark for tactile representation learning in robotic manipulation remains challenging due to the diversity of tactile sensor designs, data formats, and robot embodiments. Rather than seeking to establish such, we explore a scalable and promising direction for future development: egocentric vision paired with full-hand tactile data. To this end, we introduce \textbf{HT-Bench}, a large-scale multi-task benchmark for dexterous full-hand tactile sensing, comprising 10M RGB frames and 7.8M tactile frames collected across 226 tasks. HT-Bench evaluates tactile representations from three key perspectives: whether they encode meaningful contact geometry, whether they can align tactile observations with visual information, and whether they generalize to unseen tasks. To assess these capabilities, HT-Bench includes four tasks: fine-grained tactile similarity retrieval, masked tactile inpainting, vision-to-tactile synthesis, and multimodal tactile frame prediction. We further propose \textbf{HandTouch}, a vector-quantized vision--tactile encoder that learns tactile representations through progressive spatial, cross-modal, and temporal training. Across HT-Bench, HandTouch consistently outperforms representative tactile encoder baselines, improving Recall@5 on fine-grained tactile similarity retrieval from 74.65\% to 85.23\%, reducing RMSE on masked tactile inpainting from 0.022 to 0.010, and increasing OOD cIoU on vision-to-tactile synthesis from 0.628 to 0.705. These results demonstrate the effectiveness of HandTouch and suggest that large-scale egocentric full-hand tactile data provides a scalable basis for evaluating and advancing tactile representation learning in dexterous manipulation.

Related papers

• Language-Guided Representation Learning for Robust Cross-Sensor Material Recognition [1.441595027675735]
  We propose a language-guided distillation framework for learning sensor-robust tactile representations.<n>We construct a 39K-sample touch-language dataset with human-annotated material labels.<n>We train a tactile encoder to align sensor-specific tactile images with language embeddings in a shared semantic space.
  arXiv  Detail & Related papers  (2026-09-13T20:53:14Z)
• STAR: Sparse Tactile Representation Learning in Vision-Tactile-Language-Action Models for Dexterous Manipulation [24.489844571043808]
  Dexterous manipulation requires coordinated multi-finger control and effective tactile feedback.<n>We build a robot platform and teleoperation system to collect a 200-hour bimanual dexterous manipulation dataset.<n>We propose STAR, an integrated training recipe for vision-tactile-language-action models.
  arXiv  Detail & Related papers  (2026-09-11T07:57:24Z)
• Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor [52.33204893988409]
  We propose a visuo-tactile sensor featuring customized uniform illumination, alongside a unified perception framework.<n>The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
  arXiv  Detail & Related papers  (2026-08-25T07:25:27Z)
• AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion [60.79507611737292]
  Humans achieve stable and adaptive grasps by seamlessly integrating visual perception and tactile feedback.<n>Existing robotic grasping approaches rely on visual inputs and lack mechanisms for tactile-guided adaptation after contact.<n>We propose a unified visuo-tactile-fusion grasping framework that integrates grasp generation, feasibility prediction, and adaptive refinement.
  arXiv  Detail & Related papers  (2026-08-06T16:29:11Z)
• FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation [19.661583133282477]
  We present Feature-Extracted Latent Tactile (FELT), a learning-based framework that synthesizes per-finger pressure tactile images from RGB observations.<n>FELT uses a large frozen visual encoder and a lightweight query decoder to predict tactile signals in a single feed-forward pass.<n>Experiments on four contact-rich manipulation tasks demonstrate that both generated tactile images and latent tactile features improve policy success.
  arXiv  Detail & Related papers  (2026-07-22T19:32:42Z)
• Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning [6.509572742369941]
  We propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand.<n>We introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals.<n>Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining.<n>Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy.
  arXiv  Detail & Related papers  (2026-06-10T07:46:38Z)
• TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation [50.608989079323784]
  We propose TouchThinker, a tactile-language framework that scales tactile commonsense reasoning to the open world from both data and representation perspectives.<n>First, we construct TouchThinker-1M, a million-scale, multi-source tactile reasoning dataset covering textbf415 objects, textbf8 scenarios, and textbf7 sensor types.<n>Then, we propose action-aware modeling mechanism to improve tactile representation efficiency and enable efficient reasoning.
  arXiv  Detail & Related papers  (2026-06-10T03:58:32Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction [93.88239833545623]
  We present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset.<n>We show that tactile signals provide a compact yet powerful cue for grasp understanding.<n>We aim to advance multimodal egocentric perception, embodied learning, and contact-rich robotic manipulation.
  arXiv  Detail & Related papers  (2025-12-18T18:18:17Z)
• Tactile MNIST: Benchmarking Active Tactile Perception [19.93022179513013]
  We introduce the Tactile MNIST Benchmark Suite, an open-source, Gymnasium-compatible benchmark for tactile perception tasks.<n>Our benchmark suite offers diverse simulation scenarios, from simple toy environments all the way to complex tactile perception tasks using vision-based tactile sensors.<n>We also offer a comprehensive dataset comprising 13,500 synthetic 3D MNIST digit models and 153,600 real-world tactile samples collected from 600 3D printed digits.
  arXiv  Detail & Related papers  (2025-06-03T14:42:16Z)
• RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data [10.059624183053499]
  Visuo-tactile perception aims to understand an object's tactile properties, such as texture, softness, and rigidity.<n>We introduce RA-Touch, a retrieval-augmented framework that improves visuo-tactile perception by leveraging visual data enriched with tactile semantics.
  arXiv  Detail & Related papers  (2025-05-20T12:23:21Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.