See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2607.11498v1
• Date: Mon, 13 Jul 2026 12:50:32 GMT
• Title: See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models
• Abstract summary: Vision-language-action (VLA) models predict robot actions from visual observations and language instructions.<n>Most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined.<n>We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame.
• Score: 51.019904314336124
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language-action (VLA) models predict robot actions from visual observations and language instructions. These actions are defined in the robot's own 3D coordinate frame, yet most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined. The mismatch is benign under a fixed viewpoint, where the policy can memorize a single observation-to-action mapping, but grows harder as large-scale datasets aggregate demonstrations across diverse camera setups and the policy must generalize this mapping across viewpoints. We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot experiments, their advantage over an RGB-only policy widens when the camera is moved to a placement unseen during training.

Related papers

• From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model [72.8641426724502]
  Camera-Centric VLA (CamVLA) is a new VLA model that decouples manipulation controls from camera geometry.<n>CamVLA consistently improves success rates across diverse unseen viewpoints.
  arXiv  Detail & Related papers  (2026-07-06T17:59:59Z)
• PointAction: 3D Points as Universal Action Representations for Robot Control [42.12570897523391]
  We present PointAction, a framework that bridges video predictions to robot actions through explicit point-based 4D modeling.<n>PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing baselines in simulation, and generalizes to two real robot arms unseen during pretraining.
  arXiv  Detail & Related papers  (2026-06-02T17:30:50Z)
• Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning [51.799524981291235]
  End-to-end manipulation policies show promise for generalizable and dexterous robotic manipulation.<n>They inherit two key limitations from 2D foundation models.<n>We present a series of contributions to address these issues.
  arXiv  Detail & Related papers  (2026-06-01T14:01:11Z)
• Geometry-aware 4D Video Generation for Robot Manipulation [28.709339959536106]
  We propose a 4D video generation model that enforces multi-view 3D consistency of videos by supervising the model with cross-view pointmap alignment during training.<n>This geometric supervision enables the model to learn a shared 3D representation of the scene, allowing it to predict future video sequences from novel viewpoints.<n>Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets.
  arXiv  Detail & Related papers  (2025-07-01T18:01:41Z)
• OG-VLA: 3D-Aware Vision Language Action Model via Orthographic Image Generation [68.11862866566817]
  3D-aware policies achieve state-of-the-art performance on precise robot manipulation tasks, but struggle with generalization to unseen instructions, scenes, and objects.<n>We introduce OG-VLA, a novel architecture and learning framework that combines the generalization strengths of Vision Language Action models (VLAs) with the robustness of 3D-aware policies.
  arXiv  Detail & Related papers  (2025-06-01T22:15:45Z)
• SliceOcc: Indoor 3D Semantic Occupancy Prediction with Vertical Slice Representation [50.420711084672966]
  We present SliceOcc, an RGB camera-based model specifically tailored for indoor 3D semantic occupancy prediction.<n> Experimental results on the EmbodiedScan dataset demonstrate that SliceOcc achieves a mIoU of 15.45% across 81 indoor categories.
  arXiv  Detail & Related papers  (2025-01-28T03:41:24Z)
• Unsupervised Learning of Visual 3D Keypoints for Control [104.92063943162896]
  Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. We propose a framework to learn such a 3D geometric structure directly from images in an end-to-end unsupervised manner. These discovered 3D keypoints tend to meaningfully capture robot joints as well as object movements in a consistent manner across both time and 3D space.
  arXiv  Detail & Related papers  (2021-06-14T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.