CLAW: Learning Continuous Latent Action World Models via Adversarial Latent Regularization

• URL: http://arxiv.org/abs/2606.04130v1
• Date: Tue, 02 Jun 2026 18:40:24 GMT
• Title: CLAW: Learning Continuous Latent Action World Models via Adversarial Latent Regularization
• Abstract summary: We introduce CLAW, a framework for learning a world model jointly with continuous latent action representations directly from action-free videos.<n>We show that the resulting latent action world model supports both imitation learning from observation and goal-directed planning.
• Score: 15.232107188163747
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce CLAW, a fully end-to-end self-supervised framework for learning a world model jointly with continuous latent action representations directly from action-free videos. Our approach leverages adversarial latent regularization and diffusion-based video generation to capture structured and semantically meaningful action representations while modeling rich, predictive environment dynamics, without relying on any action labels or annotations. By simultaneously training the Latent Action Model and world model, CLAW learns to reason about how inferred actions induce environment transitions from visual observations alone. We show that the resulting latent action world model supports both imitation learning from observation and goal-directed planning. In imitation learning, latent actions extracted from raw videos enable behavior cloning. For planning, CLAW generates sequences of latent actions and maps them to executable actions to reach desired goals. Extensive experiments across diverse tasks and embodiments demonstrate that CLAW produces semantically meaningful latent action representations, supports effective action transfer, and enables planning and imitation from observation, outperforming existing methods.

Related papers

• Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning [53.77164209208635]
  We introduce WorldEcho, which probes action following over a broader action distribution using visual integrity and trajectory alignment.<n>We propose WorldSync, which strengthens action following along three axes: distributional coverage, representational grounding, and intervention-effect alignment.<n>Experiments on RoboTwin benchmarks and real-robot tasks show that WorldSync improves WorldEcho metrics and serves as a more reliable simulator for iterative policy improvement.
  arXiv  Detail & Related papers  (2026-08-25T17:59:49Z)
• DiLA: Disentangled Latent Action World Models [11.259992289079534]
  Latent Action Models (LAMs) enable the learning of world models from unlabeled video.<n>LAMs face a fundamental trade-off between action abstraction and generation fidelity.<n>We introduce DiLA, a novel Disentangled Latent Action world model that aims to resolve this trade-off via content-structure disentanglement.
  arXiv  Detail & Related papers  (2026-05-15T08:22:37Z)
• SCAR: Self-Supervised Continuous Action Representation Learning [36.917304453471864]
  We propose SCAR, a joint inverse-forward dynamics framework for learning unified action representations across embodiments from visual transitions.<n>Built on a pretrained generative backbone, SCAR uses an inverse dynamics model (IDM) to infer latent actions from latent observation pairs and a forward dynamics model (FDM) to predict future dynamics conditioned on them.<n>Experiments on the Procgen and Robotwin dataset show that the learned unified latent action representation serves as a stronger conditioning interface for world modeling than embodiment-specific raw actions.
  arXiv  Detail & Related papers  (2026-05-13T16:23:11Z)
• HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models [58.191567345416836]
  World Action Models (WAMs) have emerged as a promising paradigm for robot control by modeling physical dynamics.<n>HarmoWAM fully leverages a world model to unify predictive and reactive control, enabling both general transit and precise manipulation.<n>We construct three training-unseen test environments across six real-world robotic tasks, covering variations in background, position, and object semantics.
  arXiv  Detail & Related papers  (2026-05-11T17:59:56Z)
• From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models [14.109112325287208]
  We study the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions.<n>Our results reveal a formulation-task correspondence, whereas action-based latent actions excel at complex motor coordination.
  arXiv  Detail & Related papers  (2026-05-06T09:27:07Z)
• Factored Latent Action World Models [39.60866765151469]
  Learning latent actions from action-free video has emerged as a powerful paradigm for scaling up controllable world model learning.<n>This paper introduces Factored Latent Action Model (FLAM), a factored dynamics framework that decomposes the scene into independent factors.
  arXiv  Detail & Related papers  (2026-02-18T07:08:14Z)
• Olaf-World: Orienting Latent Actions for Video World Modeling [100.96069208914957]
  Scaling action-controllable world models is limited by the scarcity of action labels.<n>We present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video.
  arXiv  Detail & Related papers  (2026-02-10T18:58:41Z)
• CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos [73.51386721543135]
  We propose Contrastive Latent Action Pretraining (CLAP), a framework that aligns the visual latent space from videos with a proprioceptive latent space from robot trajectories.<n>CLAP maps video transitions onto a quantized, physically executable codebook.<n>We introduce a dual-formulation VLA framework offering both CLAP-NTP, an autoregressive model excelling at instruction following and object generalization, and CLAP-RF, a Rectified Flow-based policy designed for high-frequency, precise manipulation.
  arXiv  Detail & Related papers  (2026-01-07T16:26:33Z)
• Latent Action World Models for Control with Unlabeled Trajectories [8.965084673299858]
  We study world models that learn from heterogeneous data.<n>We introduce a family of latent-action world models that jointly use action-conditioned and action-free data.
  arXiv  Detail & Related papers  (2025-12-10T19:09:45Z)
• DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge [41.030494146004806]
  We propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling.<n>DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning.<n>Experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76.7% success rate on real robot tasks.
  arXiv  Detail & Related papers  (2025-07-06T16:14:29Z)
• Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning [65.85335291827086]
  This paper tries to learn and understand underlying semantic variations from distracting videos via offline-to-online latent distillation and flexible disentanglement constraints.<n>We pretrain the action-free video prediction model offline with disentanglement regularization to extract semantic knowledge from distracting videos.<n>For finetuning in the online environment, we exploit the knowledge from the pretrained model and introduce a disentanglement constraint to the world model.
  arXiv  Detail & Related papers  (2025-03-11T13:50:22Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)
• PALM: Predicting Actions through Language Models [74.10147822693791]
  We introduce PALM, an approach that tackles the task of long-term action anticipation. Our method incorporates an action recognition model to track previous action sequences and a vision-language model to articulate relevant environmental details. Our experimental results demonstrate that PALM surpasses the state-of-the-art methods in the task of long-term action anticipation.
  arXiv  Detail & Related papers  (2023-11-29T02:17:27Z)
• Diffusion Model-Augmented Behavioral Cloning [4.363232795241617]
  This work proposes an imitation learning framework that benefits from modeling both the conditional and joint probability of the expert distribution. Our proposed Diffusion Model-Augmented Behavioral Cloning (DBC) employs a diffusion model trained to model expert behaviors and learns a policy to optimize both the BC loss (conditional) and our proposed diffusion model loss (joint)
  arXiv  Detail & Related papers  (2023-02-26T15:40:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.