Geometry-Aware Motion Latents for Learning Robust Manipulation Policies

• URL: http://arxiv.org/abs/2607.04714v1
• Date: Mon, 06 Jul 2026 06:34:14 GMT
• Title: Geometry-Aware Motion Latents for Learning Robust Manipulation Policies
• Abstract summary: We introduce GeoMoLa, which learns discrete motion latent codes by predicting how point clouds evolve during manipulation.<n>GeoMoLa achieves state-of-the-art performance using only single-view RGB-D input.
• Score: 17.407684641239083
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Learning motion latents for robotic manipulation heavily relies on extracting motion patterns from visual sequences, yet effective action abstractions require understanding three-dimensional geometric transformations. Here, we introduce GeoMoLa (Geometry-Aware Motion Latents), which learns discrete motion latent codes by predicting how point clouds evolve during manipulation rather than reconstructing visual observations. This four-dimensional objective -- spatial geometry changing through time -- forces latent representations to encode actual physical motion rather than appearance patterns. GeoMoLa achieves state-of-the-art performance using only single-view RGB-D input, while existing methods require multi-view reconstruction, succeeding across diverse manipulation benchmarks. Our ablations reveal that geometric prediction is the key to driving performance, quantitatively validating that manipulation depends on spatial understanding. Furthermore, the learned codes exhibit effective motion abstraction: applying them to novel scenes produces physically consistent transformations regardless of visual context. Our real-world experiments also confirm this robustness capability, achieving robust manipulation with minimal demonstrations in cluttered environments where geometric reasoning determines success. Thus, we demonstrate that effective motion latents for robot control can better emerge from understanding motion through its three-dimensional effects rather than pixel-level patterns.

Related papers

• GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos [8.414561051327915]
  We present GeoLAM, a framework for learning geometry-grounded latent actions from action-free human videos.<n>GeoLAM combines future-frame reconstruction through a frozen geometric feature hierarchy with motion supervision from a training-only 4D geometry teacher.
  arXiv  Detail & Related papers  (2026-09-15T12:31:34Z)
• Iterative Grasp Pose Refinement: A Deep Reinforcement Learning Approach for 2D Vision [2.5899040911480182]
  This work proposes a reinforcement learning-based framework for robotic grasp refinement.<n>A geometric-based algorithm generates initial grasp candidates, which are iteratively refined by the proposed framework.<n>Experiments conducted on 300 objects from the Dex-Net dataset demonstrate the framework's effectiveness.
  arXiv  Detail & Related papers  (2026-08-18T10:44:58Z)
• ODeform: Learning Continuous 4D Motion for Shape Deformation with Neural ODEs [21.51657616248173]
  We present ODeform, a novel extension of Neural Ordinary Differential Equations to continuous 4D dynamics of deformable objects in 3D space.<n>Our method transforms 3D point clouds and physical conditions (like material properties) into a unified latent space.<n>By solving the resulting ordinary differential equations over time, we model deformations as continuous flows within this learned embedding, eliminating the need for discrete time steps while maintaining computational efficiency.
  arXiv  Detail & Related papers  (2026-07-22T19:08:31Z)
• GeoHAT: Geometry-Adaptive Hybrid Action Transformer for Mobile Manipulation [6.488530751190965]
  Whole-body mobile manipulation requires coordinating mobile base and manipulator under shifting viewpoints.<n>We present GeoHAT, an end-to-end diffusion-based framework built on a simple principle.<n>Experiments on the ManiSkill-HAB simulation benchmark demonstrate that GeoHAT achieves a 79.3% mean success rate.
  arXiv  Detail & Related papers  (2026-06-11T14:25:09Z)
• GenMatter: Perceiving Physical Objects with Generative Matter Models [50.121713214509604]
  Humans robustly segment moving entities that constitute independently moveable chunks of matter.<n>Existing computer vision systems lack a unified approach that works across diverse settings.<n>We propose a generative model that hierarchically groups low-level motion cues and high-level appearance features into particles.
  arXiv  Detail & Related papers  (2026-04-24T02:18:18Z)
• Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation [57.28703268044067]
  We propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation.<n>Our framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
  arXiv  Detail & Related papers  (2026-03-22T04:18:54Z)
• OWL: A Novel Approach to Machine Perception During Motion [0.0]
  We introduce a perception-related function, OWL, designed to address the complex challenges of 3D perception during motion.<n>It derives its values directly from two fundamental visual motion cues, with one set of cue values per point per time instant.<n> OWL offers a unified, analytical time-based approach that enhances and simplifies key perception capabilities.
  arXiv  Detail & Related papers  (2026-03-05T21:22:17Z)
• GeoMotion: Rethinking Motion Segmentation via Latent 4D Geometry [61.24189040578178]
  We propose a fully learning-based approach that directly infers moving objects from latent feature representations via attention mechanisms.<n>Our key insight is to bypass explicit correspondence estimation and instead let the model learn to implicitly disentangle object and camera motion.<n>Our approach achieves state-of-the-art motion segmentation performance with high efficiency.
  arXiv  Detail & Related papers  (2026-02-25T11:36:33Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation [52.136378691610524]
  We present DynaRend, a representation learning framework that learns 3D-aware and dynamics-informed triplane features.<n>By pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry, future dynamics, and task semantics in a unified triplane representation.<n>We evaluate DynaRend on two challenging benchmarks, RLBench and Colosseum, demonstrating substantial improvements in policy success rate, generalization to environmental perturbations, and real-world applicability across diverse manipulation tasks.
  arXiv  Detail & Related papers  (2025-10-28T10:17:11Z)
• ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation [135.10594078615952]
  We introduce ACID, an action-conditional visual dynamics model for volumetric deformable objects. A benchmark contains over 17,000 action trajectories with six types of plush toys and 78 variants. Our model achieves the best performance in geometry, correspondence, and dynamics predictions.
  arXiv  Detail & Related papers  (2022-03-14T04:56:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.