Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations

• URL: http://arxiv.org/abs/2608.29434v1
• Date: Sat, 29 Aug 2026 20:41:41 GMT
• Title: Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations
• Abstract summary: JEPA world models make latent-space planning a practical route to control.<n>With 0.3-15% of scene points moving, the slow-feature optimum of latent prediction compounds with the geometric shortcut of 3D self-supervision.<n>We lift three canonical JEPA designs to point clouds, frozen-encoder, distribution-prior, and action-sensitive, and re-sense the stable-worldmodel benchmark.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: JEPA world models make latent-space planning a practical route to control, but they are built almost exclusively on images. Whether latent prediction survives geometric observations is unclear: point clouds are sparse, unordered, and self-occluded, and with 0.3-15% of scene points moving, the slow-feature optimum of latent prediction compounds with the geometric shortcut of 3D self-supervision. We lift three canonical JEPA designs to point clouds, frozen-encoder, distribution-prior, and action-sensitive, and re-sense the stable-worldmodel benchmark so that only the observation differs from the image baselines. All three plan without collapse: the distribution-prior model is statistically equivalent to its re-evaluated image counterpart on every benchmark, and the action-sensitive model attains the strongest result in our controlled comparison where the most geometry moves. Probing explains why: object positions are almost perfectly linearly decodable and attention falls on the few moving points. Planning withstands heavy dropout never seen in training, though range noise defeats the thinnest scene. Geometry finally makes a commanded 3D target a natural goal interface: we construct the goal latent from the target and the current latent, at no cost in success rate, without a goal observation.

Related papers

• UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling [24.745715007832953]
  Joint-Embedding Predictive Architectures (JEPAs) have emerged as a principled framework for self-supervised learning of world models in compact latent spaces.<n>We present UniJEPA, a unified JEPA that jointly learns photometric prediction (image-level transformations) and temporal prediction (video-level next-state dynamics) in one shared latent space.
  arXiv  Detail & Related papers  (2026-08-07T16:55:58Z)
• J-LAW: Joint Localization and Actionable World Modeling via Coupled Latent Factor Graphs [5.78717443304847]
  Action-conditioned world models learn compact latent dynamics for planning but ignore global metric consistency.<n>We propose J-LAW: a coupled factor graph that jointly optimize metric object poses, latent world states, and latent landmark embeddings.<n>We cast observation, action-conditioned prediction, metric odometry, pose--latent coupling, latent loop closure, and latent landmark observation as probabilistic factors in a single MAP objective.
  arXiv  Detail & Related papers  (2026-06-27T03:39:44Z)
• Lifting Embodied World Models for Planning and Control [59.09016913513998]
  We train a lightweight policy that maps high-level actions to sequences of low-level joint actions.<n>We instantiate this framework for a human-like embodiment, defining the high-level action space as a small set of 2D waypoints.<n>We show that the lifted world model substantially outperforms searching directly in low-level joint space.
  arXiv  Detail & Related papers  (2026-04-28T23:59:19Z)
• Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors [47.080898117231435]
  We introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation.<n>A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy.<n>Experiments on both objects and scene scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity.
  arXiv  Detail & Related papers  (2026-03-19T11:33:27Z)
• Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation [53.09168514034483]
  Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions.<n>We propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model.<n>Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap.
  arXiv  Detail & Related papers  (2026-02-27T08:54:20Z)
• TALO: Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction [57.46712611558817]
  3D vision foundation models have shown strong generalization in reconstructing key 3D attributes from uncalibrated images through a single feed-forward pass.<n>Recent strategies align consecutive predictions by solving global transformation, yet our analysis reveals their fundamental limitations in assumption validity, local alignment scope, and robustness under noisy geometry.<n>We propose a higher-DOF and long-term alignment framework based on Thin Plate Spline, leveraging globally propagated control points to correct spatially varying inconsistencies.
  arXiv  Detail & Related papers  (2025-12-02T02:22:20Z)
• Into the Unknown: Towards using Generative Models for Sampling Priors of Environment Uncertainty for Planning in Configuration Spaces [28.37021202108478]
  Priors are vital for planning under partial observability, yet difficult to obtain in practice.<n>We present a probabilistic-based pipeline that leverages large-scale pretrained generative models to produce priors in a zero-shot manner.<n>We establish a Matterport3D benchmark of rooms partially visible through doorways, where a robot must navigate to an unobserved target object.
  arXiv  Detail & Related papers  (2025-10-13T05:08:48Z)
• Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysis [51.37795317716487]
  We propose the Adaptive Point-Prompt Tuning (APPT) method, which fine-tunes pre-trained models with a modest number of parameters.<n>We convert raw point clouds into point embeddings by aggregating local geometry to capture spatial features followed by linear layers.<n>To calibrate self-attention across source domains of any modality to 3D, we introduce a prompt generator that shares weights with the point embedding module.
  arXiv  Detail & Related papers  (2025-08-30T06:02:21Z)
• Zero-shot Inexact CAD Model Alignment from a Single Image [53.37898107159792]
  A practical approach to infer 3D scene structure from a single image is to retrieve a closely matching 3D model from a database and align it with the object in the image.<n>Existing methods rely on supervised training with images and pose annotations, which limits them to a narrow set of object categories.<n>We propose a weakly supervised 9-DoF alignment method for inexact 3D models that requires no pose annotations and generalizes to unseen categories.
  arXiv  Detail & Related papers  (2025-07-04T04:46:59Z)
• Unsupervised 3D Point Cloud Completion via Multi-view Adversarial Learning [61.14132533712537]
  We propose MAL-UPC, a framework that effectively leverages both region-level and category-specific geometric similarities to complete missing structures.<n>Our MAL-UPC does not require any 3D complete supervision and only necessitates single-view partial observations in the training set.
  arXiv  Detail & Related papers  (2024-07-13T06:53:39Z)
• PointCA: Evaluating the Robustness of 3D Point Cloud Completion Models Against Adversarial Examples [63.84378007819262]
  We propose PointCA, the first adversarial attack against 3D point cloud completion models. PointCA can generate adversarial point clouds that maintain high similarity with the original ones. We show that PointCA can cause a performance degradation from 77.9% to 16.7%, with the structure chamfer distance kept below 0.01.
  arXiv  Detail & Related papers  (2022-11-22T14:15:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.