Disentangling Visuo-Tactile Foresight: Oracle-Guided Interface Discovery for World Action Models

• URL: http://arxiv.org/abs/2608.00547v1
• Date: Sat, 01 Aug 2026 09:04:43 GMT
• Title: Disentangling Visuo-Tactile Foresight: Oracle-Guided Interface Discovery for World Action Models
• Abstract summary: Oracle Visuo-Tactile Foresight (OVTF) supplies paired RGB and tactile futures from successful trajectories verified in simulation.<n>Within OVTF, we propose Asymmetric Phase-Local Future Memory (AFM), in which visual memory reads future vision.<n>We compare AFM with Modality-Isolated Future Memory (IFM), which removes visual-to-tactile access and processes each future modality independently.
• Score: 4.240284950848863
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contact-rich manipulation remains challenging because successful control depends on physical interaction cues that are often weakly observable from vision alone. Recent tactile world action models jointly model future visual observations and tactile signals to guide action generation, but how such futures should be structured for effective use by the action expert remains underexplored. Directly studying this question with learned world action models is difficult because end-to-end behavior entangles physically invalid visual futures, unreliable predictions, inaccurate or cross-modally inconsistent tactile forecasts, and an unreadable future-to-action interface. To make this interface independently studyable, we introduce Oracle Visuo-Tactile Foresight (OVTF), a controlled framework that supplies paired RGB and tactile futures from successful trajectories verified in simulation. By fixing the future provider, OVTF isolates the interface and asks a cleaner question: if the future is successful and physically executable, what representation allows the action expert to absorb its benefit? Within OVTF, we propose Asymmetric Phase-Local Future Memory (AFM), in which visual memory reads future vision, each tactile memory jointly attends to its own tactile stream and phase-aligned future vision, and cross-tactile access is blocked. We compare AFM with Modality-Isolated Future Memory (IFM), which removes visual-to-tactile access and processes each future modality independently. Across seven tasks on the UniVTAC simulation benchmark, AFM achieves 32.0% average success, compared with 23.7% for IFM and 14.9% for UniVTAC-ACT. This controlled comparison shows that selective phase-aligned visual-tactile routing provides a more actionable future-to-action bridge than complete modality isolation.

Related papers

• UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modeling [73.35395707747456]
  Recent advances in Vision-Language-Action models have improved robotic manipulation, yet observation-to-action learning remains limited by a fundamental transition realizability gap.<n>A visually plausible predicted future observation does not necessarily correspond to a physically realizable transition.<n>A historically executable action pattern may not necessarily realize the intended transition in the current scene and therefore requires context-aware adaptation.
  arXiv  Detail & Related papers  (2026-09-10T17:45:03Z)
• ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts [24.959404195743744]
  Training-Distribution Hallucination is a recurring phenomenon in which futures conditioned on visually shifted observations hallucinate training-domain content rather than remain faithful to the current scene.<n>We propose Semantic-Temporal WAM (ST-WAM) to improve action robustness by using DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics.
  arXiv  Detail & Related papers  (2026-07-31T03:44:56Z)
• TacWAM: Anchor-Guided World Action Model with Mechanics-Aware Tactile Prediction [15.274487606802124]
  World Action Models (WAMs) combine future-state prediction with robot action generation, but existing approaches largely rely on visual futures.<n>We present TacWAM, a mechanics-aware tactile WAM that addresses this challenge in three steps.<n>We evaluate TacWAM on four real-world contact-rich manipulation tasks covering fragile grasping, sustained surface contact, and dynamic in-hand manipulation.
  arXiv  Detail & Related papers  (2026-07-30T15:47:01Z)
• τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision [43.426795127841075]
  We present, a touch-augmented Vision-Language-Action (VLA) framework that learns an action-conditioned tactile representation from future visual supervision.<n>We also introduce TacAura, a dataset of synchronized vision, proprioception, and vision-based tactile signals across four representative contact-rich manipulation tasks.
  arXiv  Detail & Related papers  (2026-07-27T14:25:15Z)
• Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention [53.84395207661823]
  We propose Tactile-WAM, a touch-aware WAM with a Tactile Asymmetric Attention Mechanism (TAAM)<n>On ManiFeel, Tactile-WAM improves the mean success rate by 38.9% overall and by 86% on contact-rich tasks.
  arXiv  Detail & Related papers  (2026-06-25T06:54:56Z)
• LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning [51.969318585152116]
  LatentPilot exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics.<n>We propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution.<n> Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene.
  arXiv  Detail & Related papers  (2026-03-31T02:21:59Z)
• VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs [47.982092015932444]
  Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence.<n>We introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a grounding signal.<n>VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning.
  arXiv  Detail & Related papers  (2026-03-24T17:45:06Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model [73.03346643967309]
  We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling.<n>FutureVLA is designed to extract joint visuomotor embeddings by first decoupling visual and motor information.<n>In the post-training stage, we employ a latent embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors.
  arXiv  Detail & Related papers  (2026-03-11T12:39:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.