WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation

• URL: http://arxiv.org/abs/2609.07002v1
• Date: Mon, 07 Sep 2026 03:48:06 GMT
• Title: WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation
• Abstract summary: Generalizable and robust in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip.<n>We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics.
• Score: 7.577131730633025
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generalizable and robust dexterous in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip from partial and noisy observations. Although recent tactile and visuotactile RL methods achieve strong in-hand rotation in controlled settings, their robustness often degrades under pose shifts, force disturbances, and object variation. We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, supervised by multimodal reconstruction and reward prediction. Rather than using the world model for latent imagination or policy optimization, WM-Craftnet uses the learned World Synesthesia Model (WSM) as recurrent task context for an asymmetric actor--critic policy. Importantly, WSM is trained to reconstruct clean depth targets from noisy depth inputs, providing a denoised geometric state for real-robot deployment. Ablations over recurrent baselines, auxiliary heads, tactile masking, and WSM modality heads show that predictive world modeling, clean-depth supervision, and tactile contact cues all shape the learned state. A WSM pretrained on nine \(z\)-axis objects serves as a reusable prior for \(49\)-object downstream policy learning. This context improves multi-object rotation, with quantitative and qualitative evidence for unseen-object, perturbation-recovery, and sim-to-real transfer.

Related papers

• Mask World Model: Predicting What Matters for Robust Robot Policy Learning [31.96162737409163]
  Mask World Model (MWM) is a generalization of large-scale video generative pre-training world models.<n>We show that MWM exhibits superior generalization capabilities and robust resilience to texture information loss.
  arXiv  Detail & Related papers  (2026-04-21T17:05:37Z)
• HAIC: Humanoid Agile Object Interaction Control via Dynamics-Aware World Model [56.4392302336014]
  We present HAIC, a framework for robust interaction across diverse object dynamics without external state estimation.<n>Our key contribution is a dynamics predictor that estimates high-order object states (velocity, acceleration) solely from proprioceptive history.<n> Experiments on a humanoid robot show HAIC achieves high success rates in agile tasks.
  arXiv  Detail & Related papers  (2026-02-12T09:34:35Z)
• DeFM: Learning Foundation Representations from Depth for Robotics [49.77188649197404]
  We present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications.<n>DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors.<n>It achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments.
  arXiv  Detail & Related papers  (2026-01-26T19:45:31Z)
• Where to Touch, How to Contact: Hierarchical RL-MPC Framework for Geometry-Aware Long-Horizon Dexterous Manipulation [15.309183490293721]
  A key challenge in contact-rich dexterous manipulation is the need to jointly reason over geometry, kinematic constraints, and intricate, nonsmooth contact dynamics.<n>We propose a hierarchical RL--MPC framework in which a high-level reinforcement learning (RL) policy predicts a contact intention.<n>We evaluate the framework on non-prehensile tasks, including geometry-generalized pushing and object 3D reorientation.
  arXiv  Detail & Related papers  (2026-01-16T01:20:15Z)
• Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation [90.90219129619344]
  This paper presents a novel R-prior-S, Recurrent Geometric-priormodal Policy with Spiking features.<n>To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases.<n>For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network.
  arXiv  Detail & Related papers  (2026-01-13T23:36:30Z)
• ByteLoom: Weaving Geometry-Consistent Human-Object Interactions through Progressive Curriculum Learning [19.292101162897975]
  We introduce ByteLoom, a framework that generates realistic HOI videos with geometrically consistent object illustration.<n>We first propose an RCM-cache mechanism that leverages Relative Coordinate Maps (RCM) as a universal representation to maintain object's geometry consistency.<n>We then design a training curriculum that enhances model capabilities in a progressive style and relaxes the demand of hand mesh.
  arXiv  Detail & Related papers  (2025-12-28T09:38:36Z)
• DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation [52.136378691610524]
  We present DynaRend, a representation learning framework that learns 3D-aware and dynamics-informed triplane features.<n>By pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry, future dynamics, and task semantics in a unified triplane representation.<n>We evaluate DynaRend on two challenging benchmarks, RLBench and Colosseum, demonstrating substantial improvements in policy success rate, generalization to environmental perturbations, and real-world applicability across diverse manipulation tasks.
  arXiv  Detail & Related papers  (2025-10-28T10:17:11Z)
• DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulation [16.863534382288705]
  We propose a novel framework that enhances action learning by jointly predicting future states and adapting to dynamics variations based on historical trajectories.<n>DyWA achieves an average success rate of 68% in real-world experiments.
  arXiv  Detail & Related papers  (2025-03-21T02:29:52Z)
• SMORE: Simultaneous Map and Object REconstruction [66.66729715211642]
  We present a method for dynamic surface reconstruction of large-scale urban scenes from LiDAR.<n>We take a holistic perspective and optimize a compositional model of a dynamic scene that decomposes the world into rigidly-moving objects and the background.
  arXiv  Detail & Related papers  (2024-06-19T23:53:31Z)
• Learning Latent Dynamic Robust Representations for World Models [9.806852421730165]
  Visual Model-Based Reinforcement Learning (MBL) promises to agent's knowledge about the underlying dynamics of the environment. Top-temporal agents such as Dreamer often struggle with visual pixel-based inputs in the presence of irrelevant noise in the observation space. We apply a-temporal masking strategy, combined with latent reconstruction, to capture endogenous task-specific aspects of the environment for world models.
  arXiv  Detail & Related papers  (2024-05-10T06:28:42Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models.<n>Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning.<n>Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)
• Continual Visual Reinforcement Learning with A Life-Long World Model [55.05017177980985]
  We present a new continual learning approach for visual dynamics modeling.<n>We first introduce the life-long world model, which learns task-specific latent dynamics.<n>Then, we address the value estimation challenge for previous tasks with the exploratory-conservative behavior learning approach.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.