CAPE: Contrastive Action-conditioned Parallel Encoding for Embodied Planning

• URL: http://arxiv.org/abs/2606.07304v1
• Date: Fri, 05 Jun 2026 14:21:44 GMT
• Title: CAPE: Contrastive Action-conditioned Parallel Encoding for Embodied Planning
• Abstract summary: Embodied agents need to predict the future consequences of candidate actions in order to plan effectively before execution.<n>We propose CAPE, a Contrastive Action-conditioned Parallel Conver framework that learns visual dynamics by distinguishing the future outcomes induced by different action sequences.
• Score: 15.559483151991813
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied agents need to predict the future consequences of candidate actions in order to plan effectively before execution. Existing visual dynamics models learn by reconstructing future visual states or rolling out dense latent representations, which spreads learning capacity across visually salient but planning-irrelevant content rather than the action-conditioned changes that drive manipulation outcomes. We propose CAPE, a Contrastive Action-conditioned Parallel Encoding framework that learns visual dynamics by distinguishing the future outcomes induced by different action sequences. Given an initial observation and a candidate action sequence, CAPE decodes the full future latent trajectory in a single forward pass and is trained with a Goal-Convergent Contrastive Objective that aligns predictions corresponding to the same future outcome while separating those corresponding to different outcomes. On real-world DROID and zero-shot transfer to RoboCasa, CAPE substantially outperforms prior baselines on future-state retrieval, offline action matching, and closed-loop planning, while notably reducing planning-time inference cost at long prediction horizons.

Related papers

• UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modeling [73.35395707747456]
  Recent advances in Vision-Language-Action models have improved robotic manipulation, yet observation-to-action learning remains limited by a fundamental transition realizability gap.<n>A visually plausible predicted future observation does not necessarily correspond to a physically realizable transition.<n>A historically executable action pattern may not necessarily realize the intended transition in the current scene and therefore requires context-aware adaptation.
  arXiv  Detail & Related papers  (2026-09-10T17:45:03Z)
• DA-WAM: Decision-Aligned Future Latents for Driving World Models [6.756366477497099]
  DA-WAM is a framework that unifies predictive representation learning, action-conditioned future modeling, and trajectory scoring under a single decision-making objective.<n>An action-conditioned predictor generates a distinct future latent state per trajectory candidate, which is then evaluated by a future-latent-conditioned factorized scorer.<n>Experiments on NAVSIM-v1 and NAVSIM-v2 demonstrate state-of-the-art performance, while ablations and diagnostic analyses validate the key components.
  arXiv  Detail & Related papers  (2026-08-19T16:33:02Z)
• Foresight Without Seeing: Latent Futures for World Action Models [31.423405133926334]
  We propose ForeWAM, a dynamics-conditioned direct-policy WAM that provides predictive context for action generation without decoding future videos.<n>We also introduce dynamics supervised by a frozen latent action teacher, encouraging the implicit future states to capture interaction-induced transitions such as object motion, contact changes, and task progress.<n>Without embodied robot data pretraining, the standard and accelerated variants of ForeWAM achieve average success rates of 96.7% and 96.9% on LIBERO, respectively.
  arXiv  Detail & Related papers  (2026-08-12T03:27:43Z)
• ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Games [6.8149466180197]
  We propose ActSWM, an action-sensitive latent world model grounded in a transition-separation principle.<n>Across step-drift analysis, closed-loop Minecraft planning, and cross-game local action recovery, ActSWM preserves larger action-dependent rollout gaps than existing baselines.
  arXiv  Detail & Related papers  (2026-07-29T09:55:54Z)
• FutureRTC: Real-Time Robot Execution with Anticipatory-Conditioned Action Chunking [51.97839311685636]
  FutureRTC is a plug-and-play adaptation framework that predicts execution-time observations and states for asynchronous VLA control without modifying the underlying policy.<n>We show that FutureRTC achieves robustness to inference delays, resulting in smoother trajectories, faster execution, and consistently higher task success rates.
  arXiv  Detail & Related papers  (2026-07-27T05:10:16Z)
• ForgeDrive: Bidirectional Cross-Conditioning for Unified Visual-Action Generation in Autonomous Driving [12.68658021226119]
  We introduce ForgeDrive, a unified autoregressive diffusion framework with visual-action cross-conditioning.<n>We find that ForgeDrive not only unifies driving simulation, planning, and visual odometry into a single model, but also outperforms existing strong planners without any post-training strategy.
  arXiv  Detail & Related papers  (2026-06-30T07:05:48Z)
• Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models [101.44987067974999]
  Coarse-to-Control is a plan-execute VLA that introduces planning in the action-token space.<n>We show that action-token planning consistently improves over direct action generation.
  arXiv  Detail & Related papers  (2026-06-05T10:01:37Z)
• IDOL: Inverse-Dynamics-Guided Future Prediction for End-to-End Autonomous Driving [6.515607168928518]
  IDOL is an inverse-dynamics-guided future prediction framework for world-model-based end-to-end planning in latent BEV space.<n>IDOL first predicts multiple future latent scene states with a BEV world model, then applies an inverse dynamics model to adjacent latent futures.<n>These inverse-dynamics-derived signals are used to optimize the planned trajectory, turning future forecasting from passive scene anticipation into actionable planning guidance.
  arXiv  Detail & Related papers  (2026-05-29T16:05:42Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning [51.969318585152116]
  LatentPilot exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics.<n>We propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution.<n> Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene.
  arXiv  Detail & Related papers  (2026-03-31T02:21:59Z)
• Ego-centric Predictive Model Conditioned on Hand Trajectories [52.531681772560724]
  In egocentric scenarios, anticipating both the next action and its visual outcome is essential for understanding human-object interactions.<n>We propose a unified two-stage predictive framework that jointly models action and visual future in egocentric scenarios.<n>Our approach is the first unified model designed to handle both egocentric human activity understanding and robotic manipulation tasks.
  arXiv  Detail & Related papers  (2025-08-27T13:09:55Z)
• From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation [30.161471749050833]
  We propose a novel end-to-end video modeling architecture that utilizes attention mechanisms, named Anticipation via Recognition and Reasoning (ARR) ARR decomposes the action anticipation task into action recognition and reasoning tasks, and effectively learns the statistical relationship between actions by next action prediction (NAP) In addition, to address the challenge of relationship modeling that requires extensive training data, we propose an innovative approach for the unsupervised pre-training of the decoder.
  arXiv  Detail & Related papers  (2024-08-05T18:38:29Z)
• Unified Recurrence Modeling for Video Action Anticipation [16.240254363118016]
  We propose a unified recurrence modeling for video action anticipation via message passing framework. Our proposed method outperforms previous works on the large-scale EPIC-Kitchen dataset.
  arXiv  Detail & Related papers  (2022-06-02T12:16:44Z)
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors [124.30562402952319]
  The ability to predict and plan into the future is fundamental for agents acting in the world. Current learning approaches for visual prediction and planning fail on long-horizon tasks. We propose a framework for visual prediction and planning that is able to overcome both of these limitations.
  arXiv  Detail & Related papers  (2020-06-23T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.