STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation

• URL: http://arxiv.org/abs/2604.26848v2
• Date: Fri, 01 May 2026 12:48:56 GMT
• Title: STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation
• Abstract summary: STARRY is a world-model-enhanced action-generation policy that aligns spatial-temporal prediction and action generation.<n>On RoboTwin 2.0, STARRY achieves 93.82% / 93.30% average success under Clean and Randomized settings.<n>These results demonstrate the effectiveness of action-centric spatial-temporal world modeling for spatially and temporally demanding robotic manipulation.
• Score: 30.24537271257149
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Robotic manipulation requires reasoning about future spatial-temporal interactions and geometric constraints, yet existing Vision-Language-Action (VLA) policies often leave predictive representation weakly coupled with action execution, causing failures in tasks requiring precise spatial-temporal coordination. We propose STARRY, a world-model-enhanced action-generation policy that aligns spatial-temporal prediction and action generation by jointly denoising future spatial-temporal latents and actions through a unified diffusion process. To bridge 2D visual tokens and 3D metric control, STARRY introduces Geometry-Aware Selective Attention Modulation (GASAM), which converts predicted depth and end-effector geometry into token-aligned weights for selective action-attention modulation. On RoboTwin 2.0, STARRY achieves 93.82% / 93.30% average success under Clean and Randomized settings across 50 bimanual tasks. Real-world experiments show that STARRY improves average success from 42.5% to 70.8% compared with $π_{0.5}$. These results demonstrate the effectiveness of action-centric spatial-temporal world modeling for spatially and temporally demanding robotic manipulation.

Related papers

• GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation [55.47130289185285]
  Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation.<n>We present textbfMethodname, a compact, policy-native extension that inserts textbfWorld State Tokens and textbfWorld Prediction Tokens directly into the VLA backbone.
  arXiv  Detail & Related papers  (2026-08-26T11:41:55Z)
• SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space [12.198811616220835]
  World Action Models (WAMs) couple action generation with prediction of future states.<n>We propose SG-WAM, a self-guided framework that learns geometry-aware action-conditioned dynamics directly in the policy-derived representation space.
  arXiv  Detail & Related papers  (2026-08-02T17:25:08Z)
• DreamTrajectory: Trajectory-Guided Action Generation with World Model Alignment for Mobile Manipulation [33.046851310859495]
  We present DreamTrajectory, a trajectory-guided framework for language-conditioned mobile manipulation.<n>On MS-HAB, trajectory guidance raises average success from 32.3% to 47.5% and test-time refinement further to 54.8%.<n>On three real-world mobile manipulation tasks, the corresponding average success rates are 63.3%, 81.7%, and 90.0%.
  arXiv  Detail & Related papers  (2026-08-02T17:02:48Z)
• Geometric Action Model for Robot Policy Learning [68.6657929619782]
  Generalist robot policies must follow user instructions while reasoning about how objects, cameras, and robot actions interact in the 3D physical world.<n>Recent vision-language-action models (VLAs) and video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models.<n>We propose the Geometric Action Model (GAM), a language-conditioned manipulation policy.
  arXiv  Detail & Related papers  (2026-06-15T17:58:03Z)
• WAM4D: Fast 4D World Action Model via Spatial Register Tokens [52.16661403052964]
  We present WAM4D, a fast 4D world action model that uses lightweight spatial register tokens as training-time future-depth readouts.<n>We show that WAM4D improves spatial consistency and achieves competitive action prediction while maintaining efficient inference.
  arXiv  Detail & Related papers  (2026-06-12T02:49:34Z)
• Sparse2Act: Learning Action-Aligned Sparse 3D Representations for Cross-Domain Robot Manipulation [52.00134393320209]
  We introduce Sparse2Act, an observation-action alignment framework for pretraining sparse point-cloud encoders.<n>Masked sparse 3D tokens are trained to organize scene features around the workspace motion paired with the observation.<n>On the LIBERO-10 benchmark, our method achieves 86.9% average success after 500 fine-tuning steps.
  arXiv  Detail & Related papers  (2026-06-10T23:56:01Z)
• AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps [7.710034405765985]
  AIM is an intent-aware unified world action model that bridges this gap via an explicit spatial interface.<n>Built on a pretrained video generation model, AIM jointly models future observations and value maps within a shared mixture-of-transformers architecture.<n>Experiments on RoboTwin 2.0 benchmark show that AIM achieves a 94.0% average success rate, significantly outperforming prior unified world action baselines.
  arXiv  Detail & Related papers  (2026-04-13T07:48:58Z)
• Learning Smooth and Robust Space Robotic Manipulation of Dynamic Target via Inter-frame Correlation [12.529803316934176]
  In microgravity environments, objects are typically free-floating, lacking the support and frictional constraints found on Earth.<n>This letter proposes a data-driven space robotic manipulation approach that integrates historical temporal information and inter-frame correlation mechanisms.
  arXiv  Detail & Related papers  (2026-03-29T06:27:30Z)
• Universal Pose Pretraining for Generalizable Vision-Language-Action Policies [83.39008378156647]
  Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency.<n>We propose Pose-VLA, a decoupled paradigm that separates VLA training into a pre-training phase for extracting universal 3D spatial priors.<n>Our framework follows a two-stage pre-training pipeline, establishing fundamental spatial grounding via poses followed by motion alignment.
  arXiv  Detail & Related papers  (2026-02-23T11:00:08Z)
• HAIC: Humanoid Agile Object Interaction Control via Dynamics-Aware World Model [56.4392302336014]
  We present HAIC, a framework for robust interaction across diverse object dynamics without external state estimation.<n>Our key contribution is a dynamics predictor that estimates high-order object states (velocity, acceleration) solely from proprioceptive history.<n> Experiments on a humanoid robot show HAIC achieves high success rates in agile tasks.
  arXiv  Detail & Related papers  (2026-02-12T09:34:35Z)
• Kinematics-Aware Diffusion Policy with Consistent 3D Observation and Action Space for Whole-Arm Robotic Manipulation [9.547155394196158]
  Whole-body control of robotic manipulators with awareness of full-arm kinematics is crucial for many manipulation scenarios.<n>The typical approach for whole-arm manipulation is to learn actions in the robot's joint space.<n>This letter proposes a kinematics-aware imitation learning framework with consistent task, observation, and action spaces.
  arXiv  Detail & Related papers  (2025-12-19T13:34:29Z)
• Real-World Robot Control by Deep Active Inference With a Temporally Hierarchical World Model [0.7284556903703034]
  Deep active inference is a framework that accounts for human goal-directed and exploratory actions.<n>We propose a novel deep active inference framework that consists of a world model, an action model, and an abstract world model.<n>We evaluate the framework on object-manipulation tasks with a real-world robot.
  arXiv  Detail & Related papers  (2025-12-01T17:41:01Z)
• VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation [54.81449795163812]
  We develop a general VLA model with 4D awareness for temporally coherent robotic manipulation.<n>We extract visual features, embed 1D time into 3D positions for 4D embeddings, and fuse them into a unified visual representation via a cross-attention mechanism.<n>Within this framework, the designed visual action jointly make robotic manipulation spatially-smooth and temporally temporally coherent.
  arXiv  Detail & Related papers  (2025-11-21T12:26:30Z)
• Ctrl-World: A Controllable Generative World Model for Robot Manipulation [53.71061464925014]
  Generalist robot policies can perform a wide range of manipulation skills.<n> evaluating and improving their ability with unfamiliar objects and instructions remains a significant challenge.<n>World models offer a promising, scalable alternative by enabling policies to rollout within imagination space.
  arXiv  Detail & Related papers  (2025-10-11T09:13:10Z)
• GWM: Towards Scalable Gaussian World Models for Robotic Manipulation [53.51622803589185]
  We propose a novel branch of world model named Gaussian World Model (GWM) for robotic manipulation.<n>At its core is a latent Diffusion Transformer (DiT) combined with a 3D variational autoencoder, enabling fine-grained scene-level future state reconstruction.<n>Both simulated and real-world experiments depict that GWM can precisely predict future scenes conditioned on diverse robot actions.
  arXiv  Detail & Related papers  (2025-08-25T02:01:09Z)
• Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoning [43.35947581429867]
  Vision-centric hierarchical embodied models have demonstrated strong potential for long-horizon robotic control.<n>Existing methods lack spatial awareness capabilities, limiting their effectiveness in bridging visual plans to actionable control in complex environments.<n>We propose Spatial Policy (SP), a unified spatial-aware visuomotor robotic manipulation framework via explicit spatial modeling and reasoning.
  arXiv  Detail & Related papers  (2025-08-21T10:24:18Z)
• OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation [50.59541802645156]
  Operational Space Control (OSC) has been used as an effective task-space controller for manipulation. We propose OSC for Adaptation and Robustness (OSCAR), a data-driven variant of OSC that compensates for modeling errors. We evaluate our method on a variety of simulated manipulation problems, and find substantial improvements over an array of controller baselines.
  arXiv  Detail & Related papers  (2021-10-02T01:21:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.