T-Rex: Tactile-Reactive Dexterous Manipulation
Abstract Overview
T-Rex is a tactile-reactive dexterous manipulation framework that combines large-scale human egocentric pre-training with tactile-grounded robot mid-training. The paper introduces a variable-rate Mixture-of-Transformer-Experts architecture with separate low-frequency visuomotor planning and high-frequency tactile refinement, supported by a spatio-temporal tactile VQ-VAE encoder. It is trained using a 100-hour tactile-synchronized bimanual teleoperation dataset covering over 200 objects and 22 motor primitives. The system is evaluated on a real-world benchmark of 12 contact-rich manipulation tasks involving delicate force control, deformable objects, and bimanual coordination.
Novelty
The work’s main novelty is the combination of a large tactile-rich dataset, a unified variable-rate Mixture-of-Transformers policy, and an asynchronous tactile refinement mechanism for dexterous manipulation. Its tactile module is distinctive in using a spatio-temporal VQ-VAE to encode high-frequency force dynamics and deformation signals while reusing cached visual-language context for fast closed-loop control.
Results
Across 12 tactile-reactive manipulation tasks, T-Rex achieves the best average success rate among the compared methods, exceeding the strongest baseline by more than 30%. Ablations indicate that tactile inputs, the proposed tactile representations, asynchronous refinement, and the full three-stage training recipe each contribute to performance, and the tactile-grounded mid-training setup improves generalization and low-data efficiency.
Key Points
- The paper introduces a 100-hour tactile-synchronized dexterous manipulation dataset with over 200 objects, 22 motor primitives, and synchronized RGB, tactile, robot-state, action, and language data.
- The T-Rex model separates slow visuomotor action planning from fast tactile action refinement through a variable-rate Mixture-of-Transformer-Experts architecture with a spatio-temporal tactile encoder.
- On a 12-task real-world benchmark, the method outperforms prior dexterous and tactile-aware baselines, with ablations supporting the importance of tactile integration, asynchronous control, and tactile-grounded mid-training.