AxisGuide: Grounding Robot Action Coordinate System in RGB Observations for Robust Visuomotor Manipulation

• URL: http://arxiv.org/abs/2606.06761v1
• Date: Thu, 04 Jun 2026 22:57:34 GMT
• Title: AxisGuide: Grounding Robot Action Coordinate System in RGB Observations for Robust Visuomotor Manipulation
• Abstract summary: Visuomotor manipulation policies trained via large-scale behavior cloning often fail to execute correct low-level actions under distribution shifts.<n>We introduce AxisGuide, a lightweight guidance method that bridges semantic scene understanding and action-coordinate interpretation.
• Score: 17.509625759060473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visuomotor manipulation policies trained via large-scale behavior cloning have achieved strong semantic scene understanding, yet often fail to reliably execute correct low-level actions under distribution shifts. For example, even in a simple pickup task with identical scene layouts, camera viewpoints, and illumination, performance can degrade substantially when the object is placed at unseen locations. We argue that this gap arises from insufficient action understanding, namely the inability to interpret the robot's base-frame action coordinate system in image space. To address this issue, we introduce AxisGuide, a lightweight guidance method that bridges semantic scene understanding and action-coordinate interpretation. Using camera parameters and end-effector poses, AxisGuide renders the robot base-frame axes in each camera view and augments RGB observations with a small set of cue channels that explicitly visualize the meaning of the +x, +y, and +z motions in image space. Extensive evaluations in both the LIBERO simulation and real-world environments demonstrate that AxisGuide yields substantial performance gains and improved generalization, highlighting the effectiveness of explicit action-coordinate cues for learning reliable and transferable generalist visuomotor policies.

Related papers

• One Demo is Worth a Thousand Trajectories: Action-View Augmentation for Visuomotor Policies [19.150827284122403]
  Visuomotor policies for manipulation have demonstrated remarkable potential in modeling complex robotic behaviors.<n>Minor alterations in the robot's initial configuration and unseen obstacles easily lead to out-of-distribution observations.<n>We introduce an effective data augmentation framework that generates visually realistic fisheye image sequences and corresponding physically feasible action trajectories.
  arXiv  Detail & Related papers  (2026-06-17T20:41:13Z)
• Trans2Occ: Voxel Occupancy Estimation and Grasp for Transparent Objects from Simulation to Reality [125.76063573012853]
  We present a framework for transparent object perception and manipulation based on single-view RGB input.<n>Our approach predicts voxel-space occupancy directly from a single image, providing a geometry-aware representation that supports downstream robotic grasping.
  arXiv  Detail & Related papers  (2026-06-01T06:59:13Z)
• Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera [3.219880761967806]
  This work focuses on a novel approach that recognizes object motion using pixel-level visual information extracted from a single RGB image.<n>To achieve stable learning in a high-DoF system composed of a robot arm equipped with a multi-fingered hand, we design a heterogeneous multi-agent reinforcement learning framework.<n>Each agent is trained cooperatively using role-specific observations and rewards, and the learned policies are successfully transferred from simulation to the real world.
  arXiv  Detail & Related papers  (2026-02-26T08:15:38Z)
• ArtiSG: Functional 3D Scene Graph Construction via Human-demonstrated Articulated Objects Manipulation [51.54082859171464]
  ArtiSG is a framework that constructs functional 3D scene graphs by encoding human demonstrations into structured robotic memory.<n>We show that ArtiSG significantly outperforms baselines in functional element recall and articulation estimation precision.
  arXiv  Detail & Related papers  (2025-12-31T13:10:40Z)
• Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration [58.4036440289082]
  Hand-object motion-capture (MoCap) offer large-scale, contact-rich demonstrations and hold promise for dexterous robotic scopes.<n>We introduce Dexplore, a unified single-loop optimization that performs repositories and tracking to learn robot control policies directly from MoCap at scale.
  arXiv  Detail & Related papers  (2025-09-11T17:59:07Z)
• Precise Action-to-Video Generation Through Visual Action Prompts [62.951609704196485]
  Action-driven video generation faces a precision-generality trade-off.<n>Agent-centric action signals provide precision at the cost of cross-domain transferability.<n>We "render" actions into precise visual prompts as domain-agnostic representations.
  arXiv  Detail & Related papers  (2025-08-18T17:12:28Z)
• Grounding Actions in Camera Space: Observation-Centric Vision-Language-Action Policy [47.51062818231493]
  We introduce the Observation-Centric VLA (OC-VLA) framework, which grounds action predictions directly in the camera observation space.<n>OC-VLA transforms end-effector poses from the robot base coordinate system into the camera coordinate system.<n>This strategy substantially improves model resilience to camera viewpoint variations.
  arXiv  Detail & Related papers  (2025-08-18T17:10:45Z)
• Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy [73.75271615101754]
  We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences.<n>Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations.<n>Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces.
  arXiv  Detail & Related papers  (2025-03-25T15:19:56Z)
• Diffusion Transformer Policy [48.50988753948537]
  We propose a large multi-modal diffusion transformer, dubbed as Diffusion Transformer Policy, to model continuous end-effector actions.<n>By leveraging the scaling capability of transformers, the proposed approach can effectively model continuous end-effector actions across large diverse robot datasets.
  arXiv  Detail & Related papers  (2024-10-21T12:43:54Z)
• Articulated Object Manipulation using Online Axis Estimation with SAM2-Based Tracking [57.942404069484134]
  Articulated object manipulation requires precise object interaction, where the object's axis must be carefully considered.<n>Previous research employed interactive perception for manipulating articulated objects, but typically, open-loop approaches often suffer from overlooking the interaction dynamics.<n>We present a closed-loop pipeline integrating interactive perception with online axis estimation from segmented 3D point clouds.
  arXiv  Detail & Related papers  (2024-09-24T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.