RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

• URL: http://arxiv.org/abs/2607.06559v1
• Date: Tue, 07 Jul 2026 17:58:15 GMT
• Title: RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
• Abstract summary: We introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image.<n>Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world manipulation tasks.
• Score: 43.55126782618712
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow, namely RGB-DF, provide a physically grounded representation that captures the underlying 4D dynamics of a scene. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to the low-level end-effector actions demanded by robotic systems, thereby narrowing the gap between world prediction and policy learning. Building on this insight, we introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate Rynn4DDataset 1.0, a massive dataset of over 254.4 million frames across egocentric human and robotic manipulation videos with high-quality pseudo-labels for depth and optical flow. We further propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of RynnWorld-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks, particularly excelling in tasks demanding spatial precision and temporal coordination.

Related papers

• PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation [51.385664546670284]
  We present PAIWorld, a framework that augments diffusion-transformer world models via three core components.<n>It achieves state-of-the-art multi-view 3D consistency on robotic benchmarks, ranking 1st on the WorldArena leaderboard and 2nd on the AgiBot-Challenge2026 leaderboard.
  arXiv  Detail & Related papers  (2026-06-16T18:23:23Z)
• Embody4D: A Generalist 4D World Model for Embodied AI [13.488596291534243]
  Embody4D is a dedicated video-to-video world model for embodied scenarios.<n>It is capable of arbitrary novel views from a monocular video.<n>It achieves state-of-the-art performance, serving as a robust world model.
  arXiv  Detail & Related papers  (2026-05-03T09:39:44Z)
• Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation [46.24300848525144]
  Kinema4D is a new action-conditioned 4D generative robotic simulator that disentangles the robot-world interaction into: i.e. precise 4D representation of robot controls, ii. Generative 4D modeling of environmental reactions.<n>For the first time, it shows potential zero-shot transfer capability, providing a high-fidelity foundation for advancing next-generation embodied simulation.
  arXiv  Detail & Related papers  (2026-03-17T15:36:38Z)
• Robo3R: Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction [70.06600045165905]
  3D spatial perception is fundamental to general robotic manipulation, yet obtaining reliable, high-quality 3D geometry remains challenging.<n>We introduce Robo3R, a manipulation-ready 3D reconstruction model that predicts accurate, metric-scale scene geometry directly from RGB images and robot states in real time.<n>We observe consistent gains in performance, suggesting the promise of this alternative 3D sensing module for robotic manipulation.
  arXiv  Detail & Related papers  (2026-02-10T18:58:15Z)
• VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control [83.92729346325163]
  VerseCrafter is a 4D-aware video world model that enables explicit and coherent control over both camera and object dynamics.<n>Our approach is centered on a novel 4D Geometric Control representation, which encodes the world state through a static background point cloud.<n>These 4D controls are rendered into conditioning signals for a pretrained video diffusion model, enabling the generation of high-fidelity, view-consistent videos.
  arXiv  Detail & Related papers  (2026-01-08T17:28:52Z)
• Any4D: Unified Feed-Forward Metric 4D Reconstruction [39.62006179006032]
  We present Any4D, a scalable multi-view transformer for metric-scale, dense feed-forward 4D reconstruction.<n>Any4D directly generates per-pixel motion and geometry predictions for N frames.<n>We achieve superior performance across diverse setups - both in terms of accuracy (2-3X lower error) and compute efficiency (15X faster)
  arXiv  Detail & Related papers  (2025-12-11T18:57:39Z)
• TesserAct: Learning 4D Embodied World Models [66.8519958275311]
  We learn a 4D world model by training on RGB-DN (RGB, Depth, and Normal) videos.<n>This not only surpasses traditional 2D models by incorporating detailed shape, configuration, and temporal changes into their predictions, but also allows us to effectively learn accurate inverse dynamic models for an embodied agent.
  arXiv  Detail & Related papers  (2025-04-29T17:59:30Z)
• 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives [115.67081491747943]
  Dynamic 3D scene representation and novel view synthesis are crucial for enabling AR/VR and metaverse applications.<n>We reformulate the reconstruction of a time-varying 3D scene as approximating its underlying 4D volume.<n>We derive several compact variants that effectively reduce the memory footprint to address its storage bottleneck.
  arXiv  Detail & Related papers  (2024-12-30T05:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.