Tactile Modality Fusion for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.14604v1
• Date: Sun, 15 Mar 2026 20:57:51 GMT
• Title: Tactile Modality Fusion for Vision-Language-Action Models
• Abstract summary: We propose TacFiLM, a lightweight modality-fusion approach that integrates visual-tactile signals into vision-language-action (VLA) models.<n>Results show consistent improvements in success rate, direct insertion performance, completion time, and force stability across both in-distribution and out-of-distribution tasks.
• Score: 22.788833830429766
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose TacFiLM, a lightweight modality-fusion approach that integrates visual-tactile signals into vision-language-action (VLA) models. While recent advances in VLA models have introduced robot policies that are both generalizable and semantically grounded, these models mainly rely on vision-based perception. Vision alone, however, cannot capture the complex interaction dynamics that occur during contact-rich manipulation, including contact forces, surface friction, compliance, and shear. While recent attempts to integrate tactile signals into VLA models often increase complexity through token concatenation or large-scale pretraining, the heavy computational demands of behavioural models necessitate more lightweight fusion strategies. To address these challenges, TacFiLM outlines a post-training finetuning approach that conditions intermediate visual features on pretrained tactile representations using feature-wise linear modulation (FiLM). Experimental results on insertion tasks demonstrate consistent improvements in success rate, direct insertion performance, completion time, and force stability across both in-distribution and out-of-distribution tasks. Together, these results support our method as an effective approach to integrating tactile signals into VLA models, improving contact-rich manipulation behaviours.

Related papers

• DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination [39.716584408571265]
  Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world.<n>We present DeCAL, a vision-language-action model that unifies understanding, imagination and action generation for contact-rich manipulation.
  arXiv  Detail & Related papers  (2026-09-08T17:47:43Z)
• τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision [43.426795127841075]
  We present, a touch-augmented Vision-Language-Action (VLA) framework that learns an action-conditioned tactile representation from future visual supervision.<n>We also introduce TacAura, a dataset of synchronized vision, proprioception, and vision-based tactile signals across four representative contact-rich manipulation tasks.
  arXiv  Detail & Related papers  (2026-07-27T14:25:15Z)
• TACO: TActile World Model as a Self-COrrector forScalable VLA Post-Training [79.94818813802682]
  We introduce TACO, a tactile-aware world-model-driven framework for scalable VLA post-training in contact-rich manipulation.<n>To incorporate tactile corrective supervision into VLA post-training, TACO combines knowledge-insulated tactile adaptation with advantage-conditioned training.
  arXiv  Detail & Related papers  (2026-07-03T00:23:30Z)
• UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models [32.87879913150572]
  We propose a unified tactile learning framework for contact-rich manipulation.<n>We show that our method improves success rate, manipulation accuracy, and contact robustness over existing methods.
  arXiv  Detail & Related papers  (2026-06-30T14:24:00Z)
• Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance [108.46436073194546]
  We argue that effective planning should be local, visually grounded, internally generated, and directly aligned with action.<n>We propose Afford-VLA, a unified framework that internalizes task-conditioned affordance as an explicit visual planning interface within VLA models.
  arXiv  Detail & Related papers  (2026-05-22T20:43:47Z)
• Modular Sensory Stream for Integrating Physical Feedback in Vision-Language-Action Models [48.57114740339529]
  MoSS is a modular sensory stream framework that adapts Vision-Language-Action models to leverage multiple sensory signals for action prediction.<n>To enable stable incorporation of new modalities, we adopt a two-stage training scheme that freezes pretrained VLA parameters in the early stage.<n>To better capture contact interaction dynamics, we incorporate an auxiliary task that predicts future physical signals.
  arXiv  Detail & Related papers  (2026-04-25T12:28:47Z)
• VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs [47.982092015932444]
  Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence.<n>We introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a grounding signal.<n>VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning.
  arXiv  Detail & Related papers  (2026-03-24T17:45:06Z)
• On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning [61.38273866396522]
  Vision-Language-Action models have emerged as a powerful paradigm for general-purpose robot learning.<n>Current methods remain unsuitable for challenging simulated- or physical-world deployments.<n>We introduce a Test-Time Reinforcement Learning for VLAs framework that enables on-the-fly policy adaptation during inference.
  arXiv  Detail & Related papers  (2026-01-11T01:51:30Z)
• Learning Generalizable Visuomotor Policy through Dynamics-Alignment [13.655111993491674]
  Recent approaches leveraging video prediction models have shown promising results by learning rich representations from large-scale datasets.<n>We propose a Dynamics-Aligned Flow Matching Policy (DAP) that integrates dynamics prediction into policy learning.<n>Our method introduces a novel architecture where policy and dynamics models provide mutual corrective feedback during action generation, enabling self-correction and improved generalization.
  arXiv  Detail & Related papers  (2025-10-31T02:29:33Z)
• Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance [63.33213516925946]
  We introduce textbfAlign-Then-stEer (textttATE), a novel, data-efficient, and plug-and-play adaptation framework.<n>Our work presents a general and lightweight solution that greatly enhances the practicality of deploying VLA models to new robotic platforms and tasks.
  arXiv  Detail & Related papers  (2025-09-02T07:51:59Z)
• VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback [21.08021535027628]
  We present VLA-Touch, an approach that enhances generalist robot policies with tactile sensing.<n>Our method introduces two key innovations: (1) a pipeline that leverages a pretrained tactile-language model that provides semantic tactile feedback for high-level task planning, and (2) a diffusion-based controller that refines VLA-generated actions with tactile signals for contact-rich manipulation.
  arXiv  Detail & Related papers  (2025-07-23T07:54:10Z)
• Learning to Fuse: Modality-Aware Adaptive Scheduling for Robust Multimodal Foundation Models [0.0]
  Modality-Aware Adaptive Fusion Scheduling (MA-AFS) learns to dynamically modulate the contribution of each modality on a per-instance basis.<n>Our work highlights the importance of adaptive fusion and opens a promising direction toward reliable and uncertainty-aware multimodal learning.
  arXiv  Detail & Related papers  (2025-06-15T05:57:45Z)
• ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation [62.58034332427291]
  ForceVLA is a novel end-to-end manipulation framework.<n>It treats external force sensing as a first-class modality within VLA systems.
  arXiv  Detail & Related papers  (2025-05-28T09:24:25Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.