Robotic Manipulation is Vision-to-Geometry Mapping ($f(v) \rightarrow G$): Vision-Geometry Backbones over Language and Video Models

• URL: http://arxiv.org/abs/2604.12908v1
• Date: Tue, 14 Apr 2026 15:57:16 GMT
• Title: Robotic Manipulation is Vision-to-Geometry Mapping ($f(v) \rightarrow G$): Vision-Geometry Backbones over Language and Video Models
• Abstract summary: We argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models.<n>We propose the Vision-Geometry-Action model, which directly conditions action generation on pretrained native 3D representations.<n> Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping.
• Score: 65.05130114320734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: At its core, robotic manipulation is a problem of vision-to-geometry mapping ($f(v) \rightarrow G$). Physical actions are fundamentally defined by geometric properties like 3D positions and spatial relationships. Consequently, we argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models. Conventional VLA and video-predictive models rely on backbones pretrained on large-scale 2D image-text or temporal pixel data. While effective, their representations are largely shaped by semantic concepts or 2D priors, which do not intrinsically align with the precise 3D geometric nature required for physical manipulation. Driven by this insight, we propose the Vision-Geometry-Action (VGA) model, which directly conditions action generation on pretrained native 3D representations. Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping that translates visual inputs directly into physical actions. To further enhance geometric consistency, we introduce a Progressive Volumetric Modulation module and adopt a joint training strategy. Extensive experiments validate the effectiveness of our approach. In simulation benchmarks, VGA outperforms top-tier VLA baselines including $π_{0.5}$ and GeoVLA, demonstrating its superiority in precise manipulation. More importantly, VGA exhibits remarkable zero-shot generalization to unseen viewpoints in real-world deployments, consistently outperforming $π_{0.5}$. These results highlight that operating on native 3D representations-rather than translating through language or 2D video priors-is a highly promising direction for achieving generalizable physical intelligence.

Related papers

• G$^3$VLA: Geometric inductive bias for Vision-Language-Action Models [3.704517635293094]
  Vision-language-action (VLA) models have made rapid progress in generalist robot manipulation.<n>Their visual tokens remain grounded in 2D image coordinates rather than the calibrated geometry of the robot's cameras.<n>We propose G$3$VLA, a camera-aware geometric module that injects calibrated structure into the visual-token stream of a pretrained VLA.
  arXiv  Detail & Related papers  (2026-06-23T12:02:36Z)
• Geometric Action Model for Robot Policy Learning [68.6657929619782]
  Generalist robot policies must follow user instructions while reasoning about how objects, cameras, and robot actions interact in the 3D physical world.<n>Recent vision-language-action models (VLAs) and video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models.<n>We propose the Geometric Action Model (GAM), a language-conditioned manipulation policy.
  arXiv  Detail & Related papers  (2026-06-15T17:58:03Z)
• Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning [51.799524981291235]
  End-to-end manipulation policies show promise for generalizable and dexterous robotic manipulation.<n>They inherit two key limitations from 2D foundation models.<n>We present a series of contributions to address these issues.
  arXiv  Detail & Related papers  (2026-06-01T14:01:11Z)
• PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction [57.63073414949329]
  We propose PointACT, a dual-system 3D-aware VLA policy that integrates hierarchical 3D point cloud representations directly into the action decoding process.<n>PointACT employs a multi-scale point-action interaction mechanism with efficient bottleneck window self-attention, enabling evolving action tokens to densely attend to both local geometric detail and global scene structure.
  arXiv  Detail & Related papers  (2026-05-20T17:10:31Z)
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding [50.098085774845195]
  We propose a paradigm shift by leveraging the implicit spatial prior within large-scale generation models.<n>We introduce GeneEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator.
  arXiv  Detail & Related papers  (2026-03-19T17:59:58Z)
• Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation [53.09168514034483]
  Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions.<n>We propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model.<n>Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap.
  arXiv  Detail & Related papers  (2026-02-27T08:54:20Z)
• Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos [39.05067965462225]
  Vision-Language-Action (VLA) models integrate visual perception with language-guided policy learning.<n>Currently, most existing approaches rely on 2D visual inputs to perform actions in 3D physical environments.<n>We propose a Spatial-Aware VLA Pretraining paradigm that performs explicit alignment between visual space and physical space during pretraining.<n>We instantiate this paradigm with VIPA-VLA, a dual-encoder architecture that incorporates a 3D visual encoder to augment semantic visual representations with 3D-aware features.
  arXiv  Detail & Related papers  (2025-12-15T08:31:47Z)
• GLaD: Geometric Latent Distillation for Vision-Language-Action Models [106.53332923530245]
  We introduce GLaD, a geometry-aware Vision-Language-Action (VLA) framework that incorporates 3D geometric priors during pretraining through knowledge distillation.<n>GLaD achieves 94.1% average success rate across four LIBERO task suites, outperforming UniVLA (92.5%) which uses identical pretraining data.
  arXiv  Detail & Related papers  (2025-12-10T13:07:27Z)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning [36.62798449863548]
  Vision-Language Models (VLMs) still lack robustness in spatial intelligence.<n>We present G$2$VLM, a vision-language model that bridges two fundamental aspects of spatial intelligence.
  arXiv  Detail & Related papers  (2025-11-26T18:59:39Z)
• Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding [11.222744122842023]
  We introduce a plug-and-play module that implicitly incorporates 3D geometry features into Vision-Language-Action (VLA) models.<n>Our method significantly improves the performance of state-of-the-art VLA models across diverse scenarios.
  arXiv  Detail & Related papers  (2025-07-01T04:05:47Z)
• OG-VLA: 3D-Aware Vision Language Action Model via Orthographic Image Generation [68.11862866566817]
  3D-aware policies achieve state-of-the-art performance on precise robot manipulation tasks, but struggle with generalization to unseen instructions, scenes, and objects.<n>We introduce OG-VLA, a novel architecture and learning framework that combines the generalization strengths of Vision Language Action models (VLAs) with the robustness of 3D-aware policies.
  arXiv  Detail & Related papers  (2025-06-01T22:15:45Z)
• Unsupervised Learning of Visual 3D Keypoints for Control [104.92063943162896]
  Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. We propose a framework to learn such a 3D geometric structure directly from images in an end-to-end unsupervised manner. These discovered 3D keypoints tend to meaningfully capture robot joints as well as object movements in a consistent manner across both time and 3D space.
  arXiv  Detail & Related papers  (2021-06-14T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.