Latent Geometry Beyond Search: Amortizing Planning in World Models

• URL: http://arxiv.org/abs/2605.08732v1
• Date: Sat, 09 May 2026 06:36:23 GMT
• Title: Latent Geometry Beyond Search: Amortizing Planning in World Models
• Abstract summary: We show that a learned representation can simplify control, rather than merely enabling prediction.<n>We replace iterative planning with a lightweight Goal-Conditioned Inverse Dynamics Model (GC-IDM)<n>Our controller matches or exceeds CEM in seven of eight environment-protocol settings while reducing per-decision cost by 100-130x.
• Score: 17.571800971505787
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern vision-based world models can represent observations as compact yet expressive latent manifolds, but fast goal-oriented planning in these spaces remains challenging. This raises a central question: when does a learned representation simplify control, rather than merely enabling prediction? We study this question in a pretrained LeWorldModel, whose latent geometry is regularized for smoothness and uniformity. Our key insight is that, under such geometry, planning can be amortized into a latent inverse-dynamics mapping instead of requiring online search. We therefore replace iterative planning with a lightweight Goal-Conditioned Inverse Dynamics Model (GC-IDM) that maps the current latent state, goal latent state, and remaining horizon directly to the next action. Empirically, across four benchmark environments spanning navigation, contact-rich manipulation, and continuous control, our controller matches or exceeds CEM in seven of eight environment-protocol settings while reducing per-decision cost by 100-130x. A broader sweep over test-time planners (CEM, MPPI, iCEM, and gradient-based methods) shows that this result is not specific to a particular optimizer. These findings suggest that much of the structure recovered by test-time planning is already locally encoded in the latent representation. More broadly, our results indicate that sufficiently structured latent spaces can shift part of the planning burden from online optimization to learned inference.

Related papers

• LePlanner: An Iterative Amortized Controller For World Models [0.0]
  We propose LePlanner, an amortized iterative controller that learns to construct and refine latent action sequences through a frozen world-model predictor.<n>LePlanner is trained with an arrival-and-hold objective that encourages the controller to reach the goal at the earliest feasible horizon and remain there.<n>It achieves success rates of 98% on PushT, 100% on Reacher, 100% on TwoRooms, and 92% on the OGBench Cube task.
  arXiv  Detail & Related papers  (2026-09-12T10:01:21Z)
• Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution [51.519325778407016]
  World models let robots imagine possible futures, but exploiting this capability for real-time control is bottlenecked by a representation misalignment.<n>We present Hydra, a discrete World Action Model that closes this gap by moving the planner, both the sampler and the evaluator, inside the model.<n> Hydra outperforms state-of-the-art world models in goal-directed planning, while matching or exceeding the closed-loop execution capabilities of leading reactive foundation policies.
  arXiv  Detail & Related papers  (2026-08-29T01:58:19Z)
• ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning [66.42476564611607]
  Diffusion models assume that depth can be recovered as a globally smooth field through iterative denoising.<n>We introduce ARDepth, which formulates depth estimation as structured auto-regressive generation.<n>We show that our approach achieves strong performance and produces structurally consistent depth predictions across scales.
  arXiv  Detail & Related papers  (2026-07-14T07:06:46Z)
• LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields [7.898784066160524]
  We present LeCropFollow, a visual navigation framework that bypasses explicit geometric modeling in favor of a learned latent representation.<n>We show that LeCropFollow matches state-of-the-art baselines in unstructured rows but significantly outperforms them in plantation gaps.<n>These results suggest that latent planning offers a robust alternative to geometric estimation for operations in heterogeneous agricultural environments.
  arXiv  Detail & Related papers  (2026-06-30T16:47:08Z)
• Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation [34.08516675160383]
  We propose a zero-shot semantic navigation framework for embodied agents.<n>Our method incrementally maintains an online Hierarchical 3D Scene Graph (HSG) over objects, zones, and regions, serving as a compact state abstraction for global planning.<n>Building on this memory, we introduce a hierarchical belief-based planning framework that fuses semantic priors with exploration evidence on the HSG, and performs finite-horizon rollouts on an HSG-based simulator to explicitly estimate the long-term expected returns of candidate macro-actions.
  arXiv  Detail & Related papers  (2026-06-30T03:01:44Z)
• Sensorimotor World Models: Perception for Action via Inverse Dynamics [59.121736411156384]
  We introduce a sensorimotor world model trained end-to-end with inverse dynamics regularization.<n>It prevents representation collapse and induces action-aligned representations.<n>It learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
  arXiv  Detail & Related papers  (2026-06-18T11:25:16Z)
• Back to Parsimonious Latents: Learning Task-Centric World Models from Visual Foundations [35.521396983843864]
  We propose a framework for turning foundation-model embeddings into compact, task-sufficient world representations.<n> TC-WM linearly projects high-dimensional visual embeddings into a compact latent as the dynamic space.<n>We show that TC-WM suffices to identify the underlying task-centric latent factors up to a simple transformation.
  arXiv  Detail & Related papers  (2026-05-25T09:21:43Z)
• Predictive but Not Plannable: RC-aux for Latent World Models [44.63243875072762]
  Reachability-Correction auxiliary (RC-aux)<n>We present Reachability-Correction auxiliary (RC-aux), a lightweight correction for this mismatch in reconstruction-free latent world models.<n>RC-aux keeps the world-model backbone unchanged and adds planning-aligned supervision along two axes.
  arXiv  Detail & Related papers  (2026-05-08T05:43:33Z)
• Hierarchical Planning with Latent World Models [49.82095442483551]
  hierarchical planning achieves higher success while requiring up to 4x less planning-time compute.<n>We demonstrate that this hierarchical approach enables zero-shot control on real-world non-greedy robotic tasks.
  arXiv  Detail & Related papers  (2026-04-03T17:32:36Z)
• In-Context Planning with Latent Temporal Abstractions [7.0210868165244875]
  I-TAP is an offline RL framework that unifies in-context adaptation with online planning in a learned temporal-abstraction space.<n>I-TAP consistently matches or outperforms strong model-free and model-conditioned offline baselines.
  arXiv  Detail & Related papers  (2026-02-21T02:29:19Z)
• Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memory [101.2076718776139]
  We propose a robust interactive world model capable of maintaining coherent visual memory over 1000+ frames in complex real-world environments.<n>We introduce a Pose-free Memory (HPMC) that distills historical latents into a fixed-budget geometric representation.<n>We also propose an Uncertainty-aware Action Labeling module that discretizes continuous motion into a tri-state logic.
  arXiv  Detail & Related papers  (2026-02-02T17:52:56Z)
• Parallel Stochastic Gradient-Based Planning for World Models [39.699893143984916]
  We propose a robust and highly parallelizable planner that leverages the differentiability of the learned world model for efficient optimization.<n>Our method treats states as optimization variables ("virtual states") with soft dynamics constraints, enabling parallel and easier optimization.<n>Our planner, which we call GRASP (GradAxed Planner), can be viewed as a valid version of a non-condensed or collocation-based optimal controller.
  arXiv  Detail & Related papers  (2026-01-31T02:57:47Z)
• Closing the Train-Test Gap in World Models for Gradient-Based Planning [64.36544881136405]
  We propose improved methods for training world models that enable efficient gradient-based planning.<n>At test time, our approach outperforms or matches the classical gradient-free cross-entropy method.
  arXiv  Detail & Related papers  (2025-12-10T18:59:45Z)
• OPUS: Occupancy Prediction Using a Sparse Set [64.60854562502523]
  We present a framework to simultaneously predict occupied locations and classes using a set of learnable queries. OPUS incorporates a suite of non-trivial strategies to enhance model performance. Our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.
  arXiv  Detail & Related papers  (2024-09-14T07:44:22Z)
• Temporal Predictive Coding For Model-Based Planning In Latent Space [80.99554006174093]
  We present an information-theoretic approach that employs temporal predictive coding to encode elements in the environment that can be predicted across time. We evaluate our model on a challenging modification of standard DMControl tasks where the background is replaced with natural videos that contain complex but irrelevant information to the planning task.
  arXiv  Detail & Related papers  (2021-06-14T04:31:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.