τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision

• URL: http://arxiv.org/abs/2607.24485v2
• Date: Thu, 30 Jul 2026 02:52:03 GMT
• Title: τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision
• Abstract summary: We present, a touch-augmented Vision-Language-Action (VLA) framework that learns an action-conditioned tactile representation from future visual supervision.<n>We also introduce TacAura, a dataset of synchronized vision, proprioception, and vision-based tactile signals across four representative contact-rich manipulation tasks.
• Score: 43.426795127841075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Incorporating tactile sensing into Vision-Language-Action (VLA) models holds promise for contact-rich manipulation, where visual observations alone often fail to capture critical cues about physical interactions. However, learning informative tactile representation while effectively adapting it to pretrained VLA models remains challenging under limited task-specific data. Existing methods either focus on instantaneous contact states or model temporal interaction dynamics using 6D wrench sequences, leaving high-dimensional tactile signals underexplored. To address these challenges, we present τ, a touch-augmented VLA framework that learns an action-conditioned spatiotemporal tactile representation from future visual supervision inspired by the Joint-Embedding Predictive Architecture (JEPA), and fuses it with vision-language features for action generation. This supervision operates in latent space and is used only during training, adding no deployment overhead. We also introduce TacAura, a dataset of synchronized vision, proprioception, and vision-based tactile signals across four representative contact-rich manipulation tasks. Experiments show that τ outperforms existing models and generalizes to unseen objects and scenes, delivering improved manipulation performance and robustness.

Related papers

• UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models [32.87879913150572]
  We propose a unified tactile learning framework for contact-rich manipulation.<n>We show that our method improves success rate, manipulation accuracy, and contact robustness over existing methods.
  arXiv  Detail & Related papers  (2026-06-30T14:24:00Z)
• TAP-VLA: Tactile Annotation Prompting for Vision Language Action Models [12.700838638661047]
  We propose a simple framework that supplies tactile feedback through visual augmentation rather than architectural change.<n>Because the architecture is untouched, the approach requires no tactile pre-training, adds negligible compute, and stays close to the pre-training distribution.<n>Across four contact-rich tasks, TAP-VLA succeeds on 78% of trials, compared to under 50% for vision-only fine-tuning and alternative tactile-fusion baselines.
  arXiv  Detail & Related papers  (2026-06-27T21:06:06Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning [51.969318585152116]
  LatentPilot exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics.<n>We propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution.<n> Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene.
  arXiv  Detail & Related papers  (2026-03-31T02:21:59Z)
• Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models [66.96421290733126]
  Vision-Language-Action (VLA) models have emerged as a promising paradigm for robotic manipulation.<n>We propose textbfDeepVision-VLA, built on a textbfVision-Language Mixture-of-Transformers (VL-MoT) framework.<n>DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively.
  arXiv  Detail & Related papers  (2026-03-16T17:59:54Z)
• Tactile Modality Fusion for Vision-Language-Action Models [22.788833830429766]
  We propose TacFiLM, a lightweight modality-fusion approach that integrates visual-tactile signals into vision-language-action (VLA) models.<n>Results show consistent improvements in success rate, direct insertion performance, completion time, and force stability across both in-distribution and out-of-distribution tasks.
  arXiv  Detail & Related papers  (2026-03-15T20:57:51Z)
• FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model [73.03346643967309]
  We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling.<n>FutureVLA is designed to extract joint visuomotor embeddings by first decoupling visual and motor information.<n>In the post-training stage, we employ a latent embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors.
  arXiv  Detail & Related papers  (2026-03-11T12:39:55Z)
• TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation [14.094740703476903]
  We introduce TaF-VLA, a framework that grounds high-dimensional tactile observations in physical interaction forces.<n>TaF-VLA policy significantly outperforms state-of-the-art tactile-vision-aligned and vision-only baselines on contact-rich tasks.
  arXiv  Detail & Related papers  (2026-01-28T07:34:41Z)
• Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation [14.221542785249524]
  We introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future.<n>Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-vision inputs.<n>To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals.
  arXiv  Detail & Related papers  (2025-12-29T21:06:33Z)
• Rethinking Visual Intelligence: Insights from Video Pretraining [75.32388528274224]
  Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems.<n>We investigate Video Diffusion Models (VDMs) as a promising direction for bridging the gap.
  arXiv  Detail & Related papers  (2025-10-28T14:12:11Z)
• Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey [13.43815806675263]
  Vision-Language-Action (VLA) models extend vision-language models to embodied control by mapping natural-language instructions and visual observations to robot actions.<n>Despite their capabilities, VLA systems face significant challenges due to their massive computational and memory demands.
  arXiv  Detail & Related papers  (2025-10-20T02:59:45Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• Data-efficient Large Vision Models through Sequential Autoregression [58.26179273091461]
  We develop an efficient, autoregression-based vision model on a limited dataset. We demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint.
  arXiv  Detail & Related papers  (2024-02-07T13:41:53Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.