Inference-time Policy Steering via Vision and Touch

• URL: http://arxiv.org/abs/2606.14981v1
• Date: Fri, 12 Jun 2026 22:03:21 GMT
• Title: Inference-time Policy Steering via Vision and Touch
• Abstract summary: Inference-time steering adapts pre-trained generative robot policies during deployment by verifying candidate actions before execution.<n>We introduce ViTaL, a visuo-tactile inference-time steering framework that formulates multimodal guidance as a bi-level optimization problem.<n>ViTaL improves overall success by 51% over the base policy, outperforms unimodal steering by at least 33%, and exceeds naive multimodal fusion by at least 20%.
• Score: 19.707194684285223
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Inference-time steering adapts pre-trained generative robot policies during deployment by verifying candidate actions before execution. While prior methods typically perform this verification only with visual observations, vision alone is often insufficient for contact-rich manipulation, where success depends on both global task progress and subtle local interactions such as contact force. We introduce ViTaL, a visuo-tactile inference-time steering framework that formulates multimodal guidance as a bi-level optimization problem. At the high level, visual sampling-and-verification performs long-horizon mode selection, deciding what behavior the robot should execute. At the low level, tactile-guided diffusion editing refines the selected action sequence over a shorter horizon to satisfy local contact requirements. To support outcome-based steering, ViTaL learns a visuo-tactile latent world model and employs semantically aligned visual and tactile verifiers, including a novel text-conditioned tactile reward that scores predicted tactile futures directly in latent space. Across three real-world contact-rich manipulation tasks, ViTaL improves overall success by 51% over the base policy, outperforms unimodal steering by at least 33%, and exceeds naive multimodal fusion by at least 20%. Website: https://yilin-wu98.github.io/vital_website.

Related papers

• $N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens [0.0]
  We present a vision-tactile-language-action (VTLA) foundation model capable of fine-grained contact-rich manipulation with tactile perception and tactile-feedback control.<n>We propose a training recipe for tactile integration consisting of visuo-tactile pre-training, staged tactile-pathway integration, and advantage-conditioned offline policy improvement.
  arXiv  Detail & Related papers  (2026-07-26T17:58:47Z)
• TouchWorld: A Predictive and Reactive Tactile Foundation Model for Dexterous Manipulation [14.063238637690597]
  Dexterous manipulation in everyday environments requires both anticipation and reaction.<n>We introduce TouchWorld, a predictive-and-reactive tactile foundation model for dexterous manipulation.
  arXiv  Detail & Related papers  (2026-07-08T11:28:13Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learning [33.49725304395789]
  We present ViTaS, a framework that incorporates both visual and tactile information to guide the behavior of an agent.<n>We demonstrate the effectiveness of our method in 12 simulated and 3 real-world environments.
  arXiv  Detail & Related papers  (2026-02-12T06:56:29Z)
• TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance [53.35296919674763]
  TouchGuide is a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space.<n>touchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy.<n>To facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system.
  arXiv  Detail & Related papers  (2026-01-28T04:22:47Z)
• ImplicitRDP: An End-to-End Visual-Force Diffusion Policy with Structural Slow-Fast Learning [52.86018040861575]
  We propose a unified end-to-end visual-force diffusion policy that integrates visual planning and reactive force control within a single network.<n>We introduce Structural Slow-Fast Learning, a mechanism utilizing causal attention to simultaneously process asynchronous visual and force tokens.<n>Experiments on contact-rich tasks demonstrate that ImplicitRDP significantly outperforms both vision-only and hierarchical baselines.
  arXiv  Detail & Related papers  (2025-12-11T18:59:46Z)
• VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback [21.08021535027628]
  We present VLA-Touch, an approach that enhances generalist robot policies with tactile sensing.<n>Our method introduces two key innovations: (1) a pipeline that leverages a pretrained tactile-language model that provides semantic tactile feedback for high-level task planning, and (2) a diffusion-based controller that refines VLA-generated actions with tactile signals for contact-rich manipulation.
  arXiv  Detail & Related papers  (2025-07-23T07:54:10Z)
• Touch begins where vision ends: Generalizable policies for contact-rich manipulation [18.195865256382334]
  We introduce VisuoTactile Local (ViTaL) policy learning, a framework that solves fine-grained manipulation tasks.<n>ViTaL achieves around 90% success on contact-rich tasks in unseen environments.
  arXiv  Detail & Related papers  (2025-06-16T17:59:48Z)
• ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers [7.505873965164197]
  We introduce ViTaPEs, a framework to learn task-agnostic representations for visuotactile perception.<n>Our approach exploits a novel multi-scale positional encoding scheme to capture intra-modal structures.<n>We show that ViTaPEs surpasses state-of-the-art baselines across various recognition tasks.
  arXiv  Detail & Related papers  (2025-05-26T14:19:29Z)
• Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation [58.95799126311524]
  Humans can accomplish contact-rich tasks using vision and touch, with highly reactive capabilities such as fast response to external changes and adaptive control of contact forces.<n>Existing visual imitation learning approaches rely on action chunking to model complex behaviors.<n>We introduce TactAR, a low-cost teleoperation system that provides real-time tactile feedback through Augmented Reality.
  arXiv  Detail & Related papers  (2025-03-04T18:58:21Z)
• ViT-Lens: Towards Omni-modal Representations [64.66508684336614]
  ViT-Lens-2 is a framework for representation learning of increasing modalities. We show that ViT-Lens-2 can learn representations for 3D point cloud, depth, audio, tactile and EEG. By seamlessly integrating ViT-Lens-2 into Multimodal Foundation Models, we enable Any-modality to Text and Image Generation.
  arXiv  Detail & Related papers  (2023-11-27T18:52:09Z)
• ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights [61.36309876889977]
  ViT-Lens enables efficient omni-modal representation learning by perceiving novel modalities with a pretrained ViT and aligning to a pre-defined space. In zero-shot 3D classification, ViT-Lens achieves substantial improvements over previous state-of-the-art. We will release the results of ViT-Lens on more modalities in the near future.
  arXiv  Detail & Related papers  (2023-08-20T07:26:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.