Latent Goal Prediction from Language for Model-Based Planning

• URL: http://arxiv.org/abs/2606.20627v1
• Date: Tue, 26 May 2026 20:04:33 GMT
• Title: Latent Goal Prediction from Language for Model-Based Planning
• Abstract summary: Latent Goal Prediction from Language (LAGO) is a framework that predicts both sequences of intermediate goal states from language instructions and action-conditioned rollouts.<n>By achieving robust and precise long-horizon planning purely from language, LAGO bridges the precision of visual goals with the flexibility of text-guided control.
• Score: 19.768973349254285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Planning with world models is bottlenecked by compounding prediction errors and the difficulty of defining optimizable goals. Visual targets provide precise local gradients but poor distant guidance, while language is flexible yet limited by noisy cross-modal alignment or dependence on large generative models unsuited for the high-sampling nature of model-based planning. To address these challenges, we introduce Latent Goal Prediction from Language (LAGO), a framework that predicts both sequences of intermediate goal states from language instructions and action-conditioned rollouts, all within the same latent space. Rather than optimizing toward a single global objective, LAGO dynamically decomposes instructions into explicitly predicted, locally tractable latent subgoals. By updating these subgoals online and using a soft minimum trajectory cost during planning, LAGO enables an agent to follow coherent latent trajectories over long horizons. Evaluation across multiple environments planning horizons shows that LAGO avoids the sharp degradation of prior methods. By achieving robust and precise long-horizon planning purely from language, LAGO bridges the precision of visual goals with the flexibility of text-guided control.

Related papers

• WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory [50.19449886428936]
  We introduce WorldScape Policy 2.0, a controllable WAM with reasoning-augmented long short-term memory.<n>The retrieved history augments perception and autoregressively generated planning tokens, yielding an implicit subgoal condition for autonomous planning.<n>Experiments in both simulation and real-world platforms demonstrate superior capabilities in long-horizon autonomous planning, fine-grained instruction following, and in-context adaptation.
  arXiv  Detail & Related papers  (2026-07-21T08:25:37Z)
• AnchorVLA: Bridging Discrete Decisions and Continuous Trajectories for Vision-Language-Action Planning [13.949868529278504]
  AnchorVLA is a hierarchical decision-anchored VLA planning framework.<n>It uses trajectory-pattern anchors as an explicit interface between high-level VLA reasoning and continuous trajectory execution.<n>It achieves a state-of-the-art Success Rate of 77.28 and a competitive Driving Score of 89.92.
  arXiv  Detail & Related papers  (2026-07-03T10:38:30Z)
• SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation [26.212246536363363]
  Object-Goal Navigation (Nav) requires embodied agents to autonomously locate specified targets using only egocentric visual observations.<n>We propose SAGE-Nav, a novel hierarchical framework that integrates the reasoning capabilities of Large Language Models (LLMs) with dynamic scene graphs.<n>We show that SAGE-Nav achieves substantial gains in navigation efficiency and zero-shot generalization while maintaining the low control latency required for physical robotic deployment.
  arXiv  Detail & Related papers  (2026-06-24T07:24:39Z)
• Beyond the Next Step: Variable-Length Latent World Models for Long-Horizon Planning [40.09669638091686]
  We propose a framework that learns to predict future latent states conditioned on action sequences of variable lengths.<n>Experiments show that Variable-length Latent World Models significantly improve latent space world models.
  arXiv  Detail & Related papers  (2026-06-19T21:50:15Z)
• Predictive but Not Plannable: RC-aux for Latent World Models [44.63243875072762]
  Reachability-Correction auxiliary (RC-aux)<n>We present Reachability-Correction auxiliary (RC-aux), a lightweight correction for this mismatch in reconstruction-free latent world models.<n>RC-aux keeps the world-model backbone unchanged and adds planning-aligned supervision along two axes.
  arXiv  Detail & Related papers  (2026-05-08T05:43:33Z)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model [53.15040805435013]
  Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
  arXiv  Detail & Related papers  (2026-03-23T17:59:42Z)
• PILOT: Planning via Internalized Latent Optimization Trajectories for Large Language Models [51.43746425777865]
  Large Language Models (LLMs) often lack the capacity to formulate global strategies, leading to error propagation in long-horizon tasks.<n>We propose PILOT, a framework designed to internalize the strategic oversight of large models into intrinsic Latent Guidance.
  arXiv  Detail & Related papers  (2026-01-07T12:38:56Z)
• NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Prediction [12.352236127154761]
  We introduce NavForesee, a novel Vision-Language Model (VLM) that unifies high-level language planning and predictive world model imagination.<n>Our approach empowers a single VLM to concurrently perform planning and predictive foresight.<n>Our work highlights the immense potential of fusing explicit language planning with implicittemporal prediction, paving the way for more intelligent and capable embodied agents.
  arXiv  Detail & Related papers  (2025-12-01T11:24:16Z)
• UnLoc: Leveraging Depth Uncertainties for Floorplan Localization [80.55849461031879]
  UnLoc is an efficient data-driven solution for sequential camera localization within floorplans.<n>We introduce a novel probabilistic model that incorporates uncertainty estimation, modeling depth predictions as explicit probability distributions.<n>We evaluate UnLoc on large-scale synthetic and real-world datasets, demonstrating significant improvements in terms of accuracy and robustness.
  arXiv  Detail & Related papers  (2025-09-14T14:45:43Z)
• Aligning LLM+PDDL Symbolic Plans with Human Objective Specifications through Evolutionary Algorithm Guidance [1.8974443450771445]
  We develop an evolutionary approach to generate a population of symbolic goal specifications.<n>We evaluate our approach on a collection of prototypical specifications in a notional naval disaster recovery task.
  arXiv  Detail & Related papers  (2024-11-30T00:58:48Z)
• Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation [64.84996994779443]
  We propose a novel Affordances-Oriented Planner for continuous vision-language navigation (VLN) task. Our AO-Planner integrates various foundation models to achieve affordances-oriented low-level motion planning and high-level decision-making. Experiments on the challenging R2R-CE and RxR-CE datasets show that AO-Planner achieves state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2024-07-08T12:52:46Z)
• Few-shot Subgoal Planning with Language Models [58.11102061150875]
  We show that language priors encoded in pre-trained language models allow us to infer fine-grained subgoal sequences. In contrast to recent methods which make strong assumptions about subgoal supervision, our experiments show that language models can infer detailed subgoal sequences without any fine-tuning.
  arXiv  Detail & Related papers  (2022-05-28T01:03:30Z)
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors [124.30562402952319]
  The ability to predict and plan into the future is fundamental for agents acting in the world. Current learning approaches for visual prediction and planning fail on long-horizon tasks. We propose a framework for visual prediction and planning that is able to overcome both of these limitations.
  arXiv  Detail & Related papers  (2020-06-23T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.