Learning Additively Compositional Latent Actions for Embodied AI

• URL: http://arxiv.org/abs/2604.03340v1
• Date: Fri, 03 Apr 2026 08:53:29 GMT
• Title: Learning Additively Compositional Latent Actions for Embodied AI
• Abstract summary: Latent action learning infers pseudo-action labels from visual transitions.<n>We introduce Additively Compositional Latent Action Model (AC-LAM)<n>AC-LAM enforces scene-wise additive composition structure over short horizons on the latent action space.
• Score: 29.26703392827422
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Latent action learning infers pseudo-action labels from visual transitions, providing an approach to leverage internet-scale video for embodied AI. However, most methods learn latent actions without structural priors that encode the additive, compositional structure of physical motion. As a result, latents often entangle irrelevant scene details or information about future observations with true state changes and miscalibrate motion magnitude. We introduce Additively Compositional Latent Action Model (AC-LAM), which enforces scene-wise additive composition structure over short horizons on the latent action space. These AC constraints encourage simple algebraic structure in the latent action space~(identity, inverse, cycle consistency) and suppress information that does not compose additively. Empirically, AC-LAM learns more structured, motion-specific, and displacement-calibrated latent actions and provides stronger supervision for downstream policy learning, outperforming state-of-the-art LAMs across simulated and real-world tabletop tasks.

Related papers

• AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning [52.47988016454267]
  Vision-Language-Action (VLA) models excel at end-to-end robotic manipulation but struggle with out-of-distribution (OOD) generalization.<n>We identify two mutually reinforcing modes: emphtrajectory overfitting, where models overfit to holistic trajectory patterns rather than compositional sub-skill semantics; and emphperceptual shortcut, where action tokens over-rely on wrist-view textures at the expense of global spatial grounding.<n>We introduce textbfAC-VLA, a plug-and-play Action Compositional learning
  arXiv  Detail & Related papers  (2026-07-17T07:51:03Z)
• Local Motion Matters: A Deconstruct-Recompose Paradigm for Reinforcement Learning Pre-training from Videos [30.202149567220356]
  Existing methods typically treat the agent as an indivisible entity, modeling motion patterns globally.<n>We propose a novel Deconstruct-compose Paradigm for learning transferable local motion representations.<n>Our method effectively transfers to diverse robotic control and manipulation tasks, significantly improving sample efficiency and performance.
  arXiv  Detail & Related papers  (2026-07-01T11:36:27Z)
• DiLA: Disentangled Latent Action World Models [11.259992289079534]
  Latent Action Models (LAMs) enable the learning of world models from unlabeled video.<n>LAMs face a fundamental trade-off between action abstraction and generation fidelity.<n>We introduce DiLA, a novel Disentangled Latent Action world model that aims to resolve this trade-off via content-structure disentanglement.
  arXiv  Detail & Related papers  (2026-05-15T08:22:37Z)
• Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models [52.17964309142613]
  Vision-Language-Action (VLA) models have advanced rapidly with stronger backbones, broader pre-training, and larger demonstration datasets.<n>We propose textbfMCF-Proto, a lightweight action head that equips VLA policies with a Motion-Centric Action Frame (MCF) and a prototype-based action parameterization.<n>Our results suggest that adding lightweight geometric and compositional structure to the action head can materially improve how VLA policies organize and generalize robotic manipulation behavior.
  arXiv  Detail & Related papers  (2026-05-12T09:03:39Z)
• InstrAct: Towards Action-Centric Understanding in Instructional Videos [12.356484522873577]
  InstrAction is a pretraining framework for instructional videos' action-centric representations.<n>We first introduce a data-driven strategy, which filters noisy captions and generates action-centric hard negatives.<n>At the visual feature level, an Action Perceiver extracts motion-relevant tokens from redundant video encodings.
  arXiv  Detail & Related papers  (2026-04-09T20:51:13Z)
• Language-Grounded Decoupled Action Representation for Robotic Manipulation [78.42228162226839]
  We propose the Language-Grounded Decoupled Action Representation (LaDA) framework to connect perception and control.<n>LaDA introduces a fine-grained intermediate layer of three interpretable action primitives--translation, rotation, and gripper control--providing explicit semantic structure for low-level actions.<n>It further employs a semantic-guided soft-label contrastive learning objective to align similar action primitives across tasks, enhancing generalization and motion consistency.
  arXiv  Detail & Related papers  (2026-03-13T13:08:26Z)
• Olaf-World: Orienting Latent Actions for Video World Modeling [100.96069208914957]
  Scaling action-controllable world models is limited by the scarcity of action labels.<n>We present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video.
  arXiv  Detail & Related papers  (2026-02-10T18:58:41Z)
• \textsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• Seeing Space and Motion: Enhancing Latent Actions with Spatial and Dynamic Awareness for VLA [21.362682837521632]
  Latent Action Models (LAMs) enable Vision- Language-Action systems to learn semantic action rep- resentations from large-scale unannotated data.<n>We propose Farsighted-LAM, a latent action framework with geometry- aware spatial encoding and multi-scale temporal modeling.<n>We further propose SSM-VLA, an end- to-end VLA framework built upon Farsighted-LAM, which integrates structured perception with a visual Chain-of-Thought module.
  arXiv  Detail & Related papers  (2025-09-30T13:41:43Z)
• CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning [47.195002937893115]
  CoMo aims to learn more informative continuous motion representations from diverse, internet-scale videos.<n>We introduce two new metrics for more robustly and affordably evaluating motion and guiding motion learning methods.<n>CoMo exhibits strong zero-shot generalization, enabling it to generate continuous pseudo actions for previously unseen video domains.
  arXiv  Detail & Related papers  (2025-05-22T17:58:27Z)
• StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning [79.44594332189018]
  Class-Incremental Learning (CIL) seeks to develop models that continuously learn new action categories over time without previously acquired knowledge.<n>Existing approaches either rely on forgetting, raising concerns over memory and privacy, or adapt static image-based methods that neglect temporal modeling.<n>We propose a unified and exemplar-free VCIL framework that explicitly disentangles and preserves information.
  arXiv  Detail & Related papers  (2025-05-20T06:46:51Z)
• Capturing Rich Behavior Representations: A Dynamic Action Semantic-Aware Graph Transformer for Video Captioning [13.411096520754507]
  Existing video captioning methods merely provide shallow or simplistic representations of object behaviors.<n>We propose a dynamic action semantic-aware graph transformer to comprehensively capture the essence of object behavior.
  arXiv  Detail & Related papers  (2025-02-19T14:16:47Z)
• Object-Centric Latent Action Learning [70.3173534658611]
  We propose a novel object-centric latent action learning framework that centers on objects rather than pixels.<n>We leverage self-supervised object-centric pretraining to disentangle action-related and distracting dynamics.<n>Our results show that object-centric pretraining mitigates the negative effects of distractors by 50%.
  arXiv  Detail & Related papers  (2025-02-13T11:27:05Z)
• Cross-Video Contextual Knowledge Exploration and Exploitation for Ambiguity Reduction in Weakly Supervised Temporal Action Localization [23.94629999419033]
  Weakly supervised temporal action localization (WSTAL) aims to localize actions in untrimmed videos using video-level labels. Our work addresses this from a novel perspective, by exploring and exploiting the cross-video contextual knowledge within the dataset. Our method outperforms the state-of-the-art methods, and can be easily plugged into other WSTAL methods.
  arXiv  Detail & Related papers  (2023-08-24T07:19:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.