Latent Actions from Factorized Transition Effects under Agent Ambiguity

• URL: http://arxiv.org/abs/2606.30544v1
• Date: Mon, 29 Jun 2026 16:45:04 GMT
• Title: Latent Actions from Factorized Transition Effects under Agent Ambiguity
• Abstract summary: We introduce Observed Transition Factorization (OTF), which decomposes each transition into a sparse set of observed transition primitives.<n>We also propose OTF-LAM, which abstracts motion primitives into action-like latents within the standard inverse-forward dynamics framework, and OTF-LAM-Dino, a decoder-free variant that predicts future states in a frozen DINOv2 representation space.
• Score: 15.872376579590437
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Latent Action Models (LAMs) learn action-like proxies from observation transitions. However, in multi-object or distractor-rich scenes, these visual effects mix agent motion with distractors, camera dynamics, and background changes, making the underlying action source ambiguous without supervision. Structuring this mixture as reusable transition effects provides an intermediate representation from which action-like latents can be more robustly formed. We introduce Observed Transition Factorization (OTF), which decomposes each transition into a sparse set of observed transition primitives. Using these primitives as the transition interface, we propose OTF-LAM, which abstracts motion primitives into action-like latents within the standard inverse-forward dynamics framework, and OTF-LAM-Dino, a decoder-free variant that predicts future states in a frozen DINOv2 representation space. Empirically, OTF primitives transfer zeroshot across controlled carrier and morphology shifts, showing reusability. Furthermore, downstream policy learning results match or outperform baselines under complex transition ambiguity.

Related papers

• UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modeling [73.35395707747456]
  Recent advances in Vision-Language-Action models have improved robotic manipulation, yet observation-to-action learning remains limited by a fundamental transition realizability gap.<n>A visually plausible predicted future observation does not necessarily correspond to a physically realizable transition.<n>A historically executable action pattern may not necessarily realize the intended transition in the current scene and therefore requires context-aware adaptation.
  arXiv  Detail & Related papers  (2026-09-10T17:45:03Z)
• DLAM: Distributional Latent Actions with Temporal Constraints [25.743937994244902]
  We introduce DLAM, a distributional latent-action model that represents each transition as a diagonal Gaussian.<n>For downstream policy learning, we freeze the encoder and train a flow-matching policy to jointly generate mean transition sequences and robot actions.<n>On held-out transitions, DLAM learns more temporally consistent latent dynamics than existing latent-action baselines and achieves stronger direct and cumulative reconstruction on held-out videos.
  arXiv  Detail & Related papers  (2026-07-29T17:09:48Z)
• DWM: Separating World Effects from Actions in Latent World Models [34.35433044964186]
  We introduce DWM (Decomposed World Model), a supervision-level framework that operationalizes this decomposition.<n>DWM augments the predictor of a latent world model with an auxiliary world head, regularized by a normalized world-contrastive objective to be action-invariant.<n>We construct W-variants of three standard control benchmarks -- PushT-W, Reacher-W, and TwoRoom-W -- each instantiating a distinct action-invariant dynamic.
  arXiv  Detail & Related papers  (2026-07-21T05:13:26Z)
• PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning [7.207198416919797]
  Latent action pretraining learns representations of visual change from pairs of observations.<n>We introduce Polar Latent Actions with Radial structure (PoLAR), which imposes a radial-direction structure on latent actions.<n>PoLAR improves downstream policy performance in simulation and real-world robot experiments.
  arXiv  Detail & Related papers  (2026-06-19T06:24:05Z)
• SCAR: Self-Supervised Continuous Action Representation Learning [36.917304453471864]
  We propose SCAR, a joint inverse-forward dynamics framework for learning unified action representations across embodiments from visual transitions.<n>Built on a pretrained generative backbone, SCAR uses an inverse dynamics model (IDM) to infer latent actions from latent observation pairs and a forward dynamics model (FDM) to predict future dynamics conditioned on them.<n>Experiments on the Procgen and Robotwin dataset show that the learned unified latent action representation serves as a stronger conditioning interface for world modeling than embodiment-specific raw actions.
  arXiv  Detail & Related papers  (2026-05-13T16:23:11Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models [31.2829405167292]
  ALAM (Algebraic Latent Action Model) turns temporal relations in action-free video into structural supervision.<n>ALAM learns latent transitions grounded by reconstruction while being regularized by composition and reversal consistency.<n>For downstream VLA learning, we freeze the pretrained encoder and use its latent transition sequences as auxiliary generative targets.
  arXiv  Detail & Related papers  (2026-05-11T16:37:07Z)
• \ extsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• Deep Compositional Phase Diffusion for Long Motion Sequence Generation [7.101824215756715]
  We present Compositional Phase Diffusion, which incorporates semantic guidance and phase details from adjacent motion clips into the diffusion process.<n> Experimental results demonstrate the competitive performance of our proposed framework in generating compositional motion sequences.
  arXiv  Detail & Related papers  (2025-10-16T08:28:46Z)
• Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy [73.75271615101754]
  We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences.<n>Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations.<n>Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces.
  arXiv  Detail & Related papers  (2025-03-25T15:19:56Z)
• Diffusion Transformer Policy [48.50988753948537]
  We propose a large multi-modal diffusion transformer, dubbed as Diffusion Transformer Policy, to model continuous end-effector actions.<n>By leveraging the scaling capability of transformers, the proposed approach can effectively model continuous end-effector actions across large diverse robot datasets.
  arXiv  Detail & Related papers  (2024-10-21T12:43:54Z)
• Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction [81.94175022575966]
  We introduce the task of action-driven human motion prediction. It aims to predict multiple plausible future motions given a sequence of action labels and a short motion history.
  arXiv  Detail & Related papers  (2022-05-31T08:38:07Z)
• Exploring Transferable and Robust Adversarial Perturbation Generation from the Perspective of Network Hierarchy [52.153866313879924]
  The transferability and robustness of adversarial examples are two practical yet important properties for black-box adversarial attacks. We propose a transferable and robust adversarial generation (TRAP) method. Our TRAP achieves impressive transferability and high robustness against certain interferences.
  arXiv  Detail & Related papers  (2021-08-16T11:52:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.