Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation

• URL: http://arxiv.org/abs/2603.01549v2
• Date: Tue, 10 Mar 2026 02:11:00 GMT
• Title: Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation
• Abstract summary: We introduce Pri4R, a simple approach that endows VLA models with an implicit understanding of world dynamics.<n>Pri4R augments VLA with a lightweight point track head that predicts 3D point tracks.<n>We show that 3D point track prediction is an effective supervision target for learning action-world dynamics.
• Score: 58.21084913574353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans learn not only how their bodies move, but also how the surrounding world responds to their actions. In contrast, while recent Vision-Language-Action (VLA) models exhibit impressive semantic understanding, they often fail to capture the spatiotemporal dynamics governing physical interaction. In this paper, we introduce Pri4R, a simple yet effective approach that endows VLA models with an implicit understanding of world dynamics by leveraging privileged 4D information during training. Specifically, Pri4R augments VLAs with a lightweight point track head that predicts 3D point tracks. By injecting VLA features into this head to jointly predict future 3D trajectories, the model learns to incorporate evolving scene geometry within its shared representation space, enabling more physically aware context for precise control. Due to its architectural simplicity, Pri4R is compatible with dominant VLA design patterns with minimal changes. During inference, we run the model using the original VLA architecture unchanged; Pri4R adds no extra inputs, outputs, or computational overhead. Across simulation and real-world evaluations, Pri4R significantly improves performance on challenging manipulation tasks, including a +10% gain on LIBERO-Long and a +40% gain on RoboCasa. We further show that 3D point track prediction is an effective supervision target for learning action-world dynamics, and validate our design choices through extensive ablations. Project page: https://jiiiisoo.github.io/Pri4R/

Related papers

• Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation [76.16065370700488]
  We introduce Lift3D-VLA, a unified VLA framework that equips models with explicit 3D point cloud reasoning and enables temporally coherent action generation.<n>We show that Lift3D-VLA achieves 10.8% and 11.1% higher mean success rates on MetaWorld and RLBench than the best-performing prior VLA methods.
  arXiv  Detail & Related papers  (2026-07-07T17:59:47Z)
• LARA: Latent Action Representation Alignment for Vision-Language-Action Models [21.4758859185298]
  Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning.<n>We propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimize LAM and VLA via representation alignment.<n>We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement.
  arXiv  Detail & Related papers  (2026-06-05T09:51:25Z)
• ELAN4D: Embodiment-Centric 4D Supervision for Vision-Language-Action Models via Plug-and-Play Adaptation [63.617951135459016]
  Vision-Language-Action (VLA) models have shown promise for robotic manipulation, yet most existing policies reactively by directly regressing actions from current observations.<n>We propose ELAN4D, an embodiment-centric, 4D-aware training framework that enhances policies with future robot key tracks as predictive-temporal supervision.
  arXiv  Detail & Related papers  (2026-05-28T19:03:30Z)
• PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction [57.63073414949329]
  We propose PointACT, a dual-system 3D-aware VLA policy that integrates hierarchical 3D point cloud representations directly into the action decoding process.<n>PointACT employs a multi-scale point-action interaction mechanism with efficient bottleneck window self-attention, enabling evolving action tokens to densely attend to both local geometric detail and global scene structure.
  arXiv  Detail & Related papers  (2026-05-20T17:10:31Z)
• ST4VLA: Spatially Guided Training for Vision-Language-Action Models [80.35847468618276]
  Large vision-language models (VLMs) excel at multimodal understanding but fall short when extended to embodied tasks.<n>We introduce ST4VLA, a dual-system Vision-Language-Action framework to align action learning with spatial priors inVLMs.<n> Empirically, ST4VLA achieves substantial improvements over vanilla VLA, with performance increasing from 66.1 -> 84.6 on Google Robot and from 54.7 -> 73.2 on WidowX Robot.
  arXiv  Detail & Related papers  (2026-02-10T18:59:17Z)
• IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance [35.90972175891154]
  Many Vision-Language-Action (VLA) models flatten image patches into a 1D token sequence, weakening the 2D spatial cues needed for precise manipulation.<n>We introduce a lightweight, training-free method that improves spatial understanding by exploiting affinity hints already available in the model's built-in vision.
  arXiv  Detail & Related papers  (2026-01-22T18:57:13Z)
• Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models [79.18306680174011]
  DSR Suite bridges gap across aspects of dataset, benchmark and model.<n>We propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR.<n>The pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories.
  arXiv  Detail & Related papers  (2025-12-23T17:56:36Z)
• Robotic VLA Benefits from Joint Learning with Motion Image Diffusion [114.60268819583017]
  Vision-Language-Action (VLA) models have achieved remarkable progress in robotic manipulation by mapping multimodal observations and instructions directly to actions.<n>We propose joint learning with motion image diffusion, a novel strategy that enhances VLA models with motion reasoning capabilities.<n> Experiments in both simulation and real-world environments demonstrate that joint learning with motion image diffusion improves the success rate of pi-series VLAs to 97.5%.
  arXiv  Detail & Related papers  (2025-12-19T19:07:53Z)
• SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead [56.74139420555097]
  We propose SwiftVLA, an architecture that enhances a compact model with 4D understanding while preserving design efficiency.<n>Our approach features a pretrained 4D visual geometry transformer with a temporal cache that extracts 4D features from 2D images.<n>Experiments in real and simulated environments show that SwiftVLA outperforms lightweight baselines and rivals VLAs up to 7 times larger.
  arXiv  Detail & Related papers  (2025-11-30T14:10:28Z)
• VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation [54.81449795163812]
  We develop a general VLA model with 4D awareness for temporally coherent robotic manipulation.<n>We extract visual features, embed 1D time into 3D positions for 4D embeddings, and fuse them into a unified visual representation via a cross-attention mechanism.<n>Within this framework, the designed visual action jointly make robotic manipulation spatially-smooth and temporally temporally coherent.
  arXiv  Detail & Related papers  (2025-11-21T12:26:30Z)
• ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver [35.25196177784228]
  We propose ReconVLA, a reconstructive VLA model with an implicit grounding paradigm.<n>Conditioned on the model's visual outputs, a diffusion transformer reconstructs the gaze region of the image.<n>This process prompts the VLA model to learn fine-grained representations and accurately allocate visual attention.
  arXiv  Detail & Related papers  (2025-08-14T04:20:19Z)
• CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning [7.780242426487376]
  We propose Chunked RL, a novel reinforcement learning framework for Vision-Language-Action (VLA) models.<n>Within this framework, we extend temporal difference (TD) learning to incorporate action chunking, a prominent characteristic of VLA models.<n>We then propose CO-RFT, an algorithm aimed at fine-tuning VLA models using a limited set of demonstrations.
  arXiv  Detail & Related papers  (2025-08-04T09:11:48Z)
• DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge [41.030494146004806]
  We propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling.<n>DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning.<n>Experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76.7% success rate on real robot tasks.
  arXiv  Detail & Related papers  (2025-07-06T16:14:29Z)
• Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding [11.222744122842023]
  We introduce a plug-and-play module that implicitly incorporates 3D geometry features into Vision-Language-Action (VLA) models.<n>Our method significantly improves the performance of state-of-the-art VLA models across diverse scenarios.
  arXiv  Detail & Related papers  (2025-07-01T04:05:47Z)
• Easi3R: Estimating Disentangled Motion from DUSt3R Without Training [69.51086319339662]
  We introduce Easi3R, a simple yet efficient training-free method for 4D reconstruction.<n>Our approach applies attention adaptation during inference, eliminating the need for from-scratch pre-training or network fine-tuning.<n>Our experiments on real-world dynamic videos demonstrate that our lightweight attention adaptation significantly outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-31T17:59:58Z)
• 3D-VLA: A 3D Vision-Language-Action Generative World Model [68.0388311799959]
  Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. We propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities in embodied environments.
  arXiv  Detail & Related papers  (2024-03-14T17:58:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.