Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation

• URL: http://arxiv.org/abs/2607.11167v1
• Date: Mon, 13 Jul 2026 07:03:29 GMT
• Title: Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation
• Abstract summary: Representing manipulation actions as 2D trajectories in the camera plane provides a compact and interpretable basis for learning complex 3D manipulation policies.<n>We propose Pix2Act, an imitation learning method that addresses these challenges by generating continuous image-space keypoint trajectories in each camera plane.<n>This reformulates high-dimensional 3D control as a simpler, more learnable 2D prediction problem.
• Score: 42.33906056027883
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Representing manipulation actions as 2D trajectories in the camera plane provides a compact and interpretable basis for learning complex 3D manipulation policies. However, it also creates challenges from out-of-frame trajectories and limited precision. We propose Pix2Act, an imitation learning method that addresses these challenges by generating continuous image-space keypoint trajectories in each camera plane and losslessly recovering end-effector poses via triangulation. This reformulates high-dimensional 3D control as a simpler, more learnable 2D prediction problem. Crucially, it aligns observations and actions in the same coordinate space, enabling equivariant transformations to jointly rotate individual camera images together with their image-space actions. We analyze the symmetry properties of this augmentation and design a network architecture that can fuse multiple camera views while respecting their per-view rotations. As a result, Pix2Act implicitly enlarges the support of the data distribution and learns invariant action structures across transformations, yielding improved generalization and overall performance. Across diverse simulated and real-world manipulation tasks, Pix2Act outperforms state-of-the-art baselines and remains robust under camera perturbations.

Related papers

• SpatialDiff: 3D-Aware Object Movement via Implicit Spatial Modeling [60.4038189368113]
  We introduce Implicit 3D Spatial Modeling, which introduces 3D prior knowledge and enables the model to internally build a comprehensive understanding of the three-dimensional spatial structure.<n>We also present Global Spatial Supervision, which constrains the latent spatial features to enable the model to perceive changes in object spatial positions caused by editing operations.<n> Experimental results demonstrate that our method significantly improves the accuracy and fidelity of spatial movement in complex scenes.
  arXiv  Detail & Related papers  (2026-08-22T04:46:08Z)
• Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning [51.799524981291235]
  End-to-end manipulation policies show promise for generalizable and dexterous robotic manipulation.<n>They inherit two key limitations from 2D foundation models.<n>We present a series of contributions to address these issues.
  arXiv  Detail & Related papers  (2026-06-01T14:01:11Z)
• Ctrl&Shift: High-Quality Geometry-Aware Object Manipulation in Visual Generation [34.92056161129864]
  We present Ctrl&Shift, an end-to-end diffusion framework to achieve geometry-consistent object manipulation without explicit 3D representations.<n>Our key insight is to decompose manipulation into two stages, object removal and reference-guided inpainting under explicit camera pose control, and encode both within a unified diffusion process.<n>To our knowledge, this is the first framework to unify fine-grained geometric control and real-world generalization for object manipulation, without relying on any explicit 3D modeling.
  arXiv  Detail & Related papers  (2026-02-11T23:36:30Z)
• Towards Rotation-only Imaging Geometry: Rotation Estimation [11.806182001858454]
  Structure from Motion (SfM) is a critical task in computer vision, aiming to recover the 3D scene structure and camera motion from a sequence of 2D images.<n>Recent pose-only imaging geometry decouples 3D coordinates from camera poses and demonstrates significantly better SfM performance through pose adjustment.
  arXiv  Detail & Related papers  (2025-11-16T02:04:32Z)
• XYZCylinder: Feedforward Reconstruction for Driving Scenes Based on A Unified Cylinder Lifting Method [27.213339282749885]
  We propose textbfXYZ Cylinder, a feedforward model based on a unified cylinder lifting method.<n>Specifically, we design a Unified Cylinder Camera Modeling (UCCM) strategy, which avoids the learning of viewpoint-dependent spatial correspondence.<n>To improve the reconstruction accuracy, we propose a hybrid representation with several dedicated modules based on newly designed Cylinder Plane Feature Group.
  arXiv  Detail & Related papers  (2025-10-09T06:58:03Z)
• LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors [107.83398512719981]
  Single-image 3D reconstruction remains a fundamental challenge in computer vision.<n>Recent advances in Latent Video Diffusion Models offer promising 3D priors learned from large-scale video data.<n>We propose LiftImage3D, a framework that effectively releases LVDMs' generative priors while ensuring 3D consistency.
  arXiv  Detail & Related papers  (2024-12-12T18:58:42Z)
• RelPose++: Recovering 6D Poses from Sparse-view Observations [66.6922660401558]
  We address the task of estimating 6D camera poses from sparse-view image sets (2-8 images) We build on the recent RelPose framework which learns a network that infers distributions over relative rotations over image pairs. Our final system results in large improvements in 6D pose prediction over prior art on both seen and unseen object categories.
  arXiv  Detail & Related papers  (2023-05-08T17:59:58Z)
• A Pose-only Solution to Visual Reconstruction and Navigation [23.86386627769292]
  Large-scale scenes and critical camera motions are great challenges facing the research community to achieve this goal. We raised a pose-only imaging geometry framework and algorithms that can help solve these challenges.
  arXiv  Detail & Related papers  (2021-03-02T07:21:08Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.