WLA$^3$: World Latent Action Modeling for Semantics, Dynamics, and Kinematics

• URL: http://arxiv.org/abs/2609.15870v1
• Date: Mon, 14 Sep 2026 16:59:00 GMT
• Title: WLA$^3$: World Latent Action Modeling for Semantics, Dynamics, and Kinematics
• Abstract summary: We introduce WLA$3$ (World Latent Action Modeling for Semantics, Dynamics, and Kinematics), a unified generalist policy model framework.<n> WLA$3$ reuses representations learned by a World Latent Action Model (WLAM) across semantics, dynamics, and kinematics.<n>On LARYBench, the final 32D latent action reaches 67.89% average classification accuracy.
• Score: 33.75014442121971
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling generalist policy models with heterogeneous data is limited by the lack of unified, low-noise action supervision. Human egocentric videos are abundant, but only a small fraction comes with high-quality hand-action labels. Observed world transitions offer a common source of action-related supervision across data sources. We introduce WLA$^3$ (World Latent Action Modeling for Semantics, Dynamics, and Kinematics), a unified generalist policy model framework built around representations learned by a World Latent Action Model (WLAM). WLAM first learns how multimodal world states change over a local interval, encoding synchronized camera views and available embodiment-state changes into a compact local latent action and a richer transition feature. Reconstruction from partial modalities and consistency across overlapping windows encourage robust transition representations. WLA$^3$ reuses them across semantics, dynamics, and kinematics: local latent actions support action-sensitive physical-dynamics modeling, segment-level features directly supervise the VLM through a Semantic Latent Aggregate (SLA), and an action expert jointly predicts latent actions together with embodiment-specific robot controls. Human videos provide scalable transition supervision, while robot trajectories ground the shared representation in executable native controls. On LARYBench, the final 32D latent action reaches 67.89\% average classification accuracy. WLA$^3$ achieves 81.9% average success across six real-robot tasks versus 66.2% for $π_{0.5}$. Performance improves as generalist policy model mid-training data scales, and human videos support human-to-robot transfer. Project page can be found at https://wla-3.github.io/.

Related papers

• Hydra-0: Action Flow for Generalist World Modeling and Control [64.88849745875753]
  Hydra-0 is a world model conditioned on action flow, which represents robot actions as pixel motion.<n>Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error.<n>A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations.
  arXiv  Detail & Related papers  (2026-08-18T17:59:30Z)
• ABot-M0.5: Unified Mobility-and-Manipulation World Action Model [27.262271208923995]
  ABot-M0.5 is built on the insight that mobile manipulation requires alignment at three levels: temporal granularity, action space, and train-test consistency.<n> Experiments on challenging mobile and fine-grained manipulation benchmarks demonstrate that ABot-M0.5 achieves state-of-the-art performance in both long-horizon task success and finegrained control accuracy.
  arXiv  Detail & Related papers  (2026-07-01T09:21:20Z)
• MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulation [17.392786435444062]
  We present MotionWAM, a real-time WAM that drives autonomous humanoid loco-manipulation from a single egocentric camera.<n>A three-stage learning framework progressively adapts the video world model to egocentric visual dynamics.
  arXiv  Detail & Related papers  (2026-06-08T08:50:14Z)
• From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data [71.22409934108924]
  Human videos are abundant and capture rich interactions, providing diverse semantic and physical cues for real-world manipulation.<n>This survey provides a unified view of how human videos are transformed into effective knowledge for Vision-Language-Action (VLA) models.<n>We highlight three key open challenges in this area: structuring unstructured videos into training-ready episodes, grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity, and designing evaluation protocols that better predict real-world deployment performance and transfer efficiency.
  arXiv  Detail & Related papers  (2026-05-18T06:19:16Z)
• RotVLA: Rotational Latent Action for Vision-Language-Action Model [54.22746299071677]
  We introduce RotVLA, a VLA framework built on a continuous rotational latent action representation.<n>Latent actions are modeled as elements of SO(n), providing continuity, compositionality, and structured geometry aligned with real-world action dynamics.<n>RotVLA consists of a VLM backbone and a flow-matching action head, pretrained on large-scale cross-embodiment robotic datasets and human videos.
  arXiv  Detail & Related papers  (2026-05-13T11:58:02Z)
• MotuBrain: An Advanced World Action Model for Robot Control [23.733029557644354]
  We present MotuBrain, a unified World Action Model that jointly models video and action under a UniDiffuser formulation.<n>A single model supports policy learning, world modeling, video generation, inverse dynamics, and joint video-action prediction.<n>Building on Motus, MotuBrain further introduces unified multiview modeling, an independent text stream for stronger language-action coupling.<n>Our inference stack combines step reduction, compilation, FP8 quantization, DiT caching, V2A-style action-only inference, and real-time chunked closed-loop execution.
  arXiv  Detail & Related papers  (2026-04-30T12:34:44Z)
• TC-IDM: Grounding Video Generation for Executable Zero-shot Robot Motion [38.88993836311246]
  Generative world models offer a promising alternative for general-purpose embodied AI.<n>We propose the Tool-Centric Inverse Dynamics Model (TC-IDM) to bridge the gap between visual planning and physical control.<n>In real-world evaluations, the world model with TC-IDM achieves an average success rate of 61.11 percent, with 77.7 percent on simple tasks and 38.46 percent on zero-shot deformable object tasks.
  arXiv  Detail & Related papers  (2026-01-26T10:06:56Z)
• FLARE: Robot Learning with Implicit World Modeling [87.81846091038676]
  $textbfFLARE$ integrates predictive latent world modeling into robot policy learning.<n>$textbfFLARE$ achieves state-of-the-art performance, outperforming prior policy learning baselines by up to 26%.<n>Our results establish $textbfFLARE$ as a general and scalable approach for combining implicit world modeling with high-frequency robotic control.
  arXiv  Detail & Related papers  (2025-05-21T15:33:27Z)
• Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy [73.75271615101754]
  We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences.<n>Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations.<n>Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces.
  arXiv  Detail & Related papers  (2025-03-25T15:19:56Z)
• HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation [54.03004125910057]
  We show that hierarchical vision-language-action models can be more effective in utilizing off-domain data than standard monolithic VLA models.<n>We show that, with the hierarchical design, the high-level VLM can transfer across significant domain gaps between the off-domain finetuning data and real-robot testing scenarios.
  arXiv  Detail & Related papers  (2025-02-08T07:50:22Z)
• Diffusion Transformer Policy [48.50988753948537]
  We propose a large multi-modal diffusion transformer, dubbed as Diffusion Transformer Policy, to model continuous end-effector actions.<n>By leveraging the scaling capability of transformers, the proposed approach can effectively model continuous end-effector actions across large diverse robot datasets.
  arXiv  Detail & Related papers  (2024-10-21T12:43:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.