Sparse2Act: Learning Action-Aligned Sparse 3D Representations for Cross-Domain Robot Manipulation

• URL: http://arxiv.org/abs/2606.12759v1
• Date: Wed, 10 Jun 2026 23:56:01 GMT
• Title: Sparse2Act: Learning Action-Aligned Sparse 3D Representations for Cross-Domain Robot Manipulation
• Abstract summary: We introduce Sparse2Act, an observation-action alignment framework for pretraining sparse point-cloud encoders.<n>Masked sparse 3D tokens are trained to organize scene features around the workspace motion paired with the observation.<n>On the LIBERO-10 benchmark, our method achieves 86.9% average success after 500 fine-tuning steps.
• Score: 52.00134393320209
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Explicit 3D representations are attractive for manipulation because they expose object shape, workspace geometry, and robot-object relations in metric coordinates. However, sparse 3D encoders are often learned through downstream task objectives, tying the representation to a particular data distribution, policy architecture, and action parameterization. We introduce Sparse2Act, an observation-action alignment framework for pretraining sparse point-cloud encoders. The key idea is to use task-space end-effector actions as geometric supervision: masked sparse 3D tokens are trained to organize scene features around the workspace motion paired with the observation. After pretraining, only the encoder initialization is reused by downstream policies, allowing them to retain their own architectures and action spaces, including joint-space commands. On the LIBERO-10 benchmark, our method achieves 86.9% average success after 500 fine-tuning steps. The same pretrained encoder supports LIBERO-to-Meta-World cross-domain transfer, achieving 73.4% average success on the Meta-World-5 benchmark. Ablations on the objective and decoder capacity show that the gains come from the masked action-alignment signal and remain useful across downstream action decoders. In real-world experiments, simulation pretraining followed by limited real-data fine-tuning achieves an average success rate of 72.5% across four tasks, demonstrating effective sim-to-real transfer. These results suggest that robot actions can provide compact geometric supervision for reusable sparse 3D representations.

Related papers

• GeomVLA: Unifying Scene, Motion, and Action in 3D [19.277341680583834]
  We present GeomVLA, a model that unifies perception, latent scene motion prediction, and action generation within a shared robot-centric 3D coordinate frame.<n>Our approach lifts pretrained VLM features into spatially grounded 3D scene tokens using depth and camera calibration.<n>We introduce a 3D Scene Trajectory Denoiser, a task-conditioned module that learns a latent representation of how scene points are expected to move in 3D.
  arXiv  Detail & Related papers  (2026-09-12T08:51:52Z)
• WAM4D: Fast 4D World Action Model via Spatial Register Tokens [52.16661403052964]
  We present WAM4D, a fast 4D world action model that uses lightweight spatial register tokens as training-time future-depth readouts.<n>We show that WAM4D improves spatial consistency and achieves competitive action prediction while maintaining efficient inference.
  arXiv  Detail & Related papers  (2026-06-12T02:49:34Z)
• GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation [54.671815855499034]
  Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors to action generation.<n>Standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution.<n>We propose textbfGaussianDream, a feed-forward 3D Gaussian world-model plug-in.
  arXiv  Detail & Related papers  (2026-05-20T05:51:30Z)
• VolumeDP: Modeling Volumetric Representation for Manipulation Policy Learning [29.62906091681386]
  VolumeDP is a policy architecture that restores spatial alignment by explicitly reasoning in 3D.<n>It achieves a state-of-the-art average success rate of 88.8% on the LIBERO simulation benchmark.
  arXiv  Detail & Related papers  (2026-03-18T13:40:24Z)
• Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation [53.09168514034483]
  Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions.<n>We propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model.<n>Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap.
  arXiv  Detail & Related papers  (2026-02-27T08:54:20Z)
• Articulated 3D Scene Graphs for Open-World Mobile Manipulation [55.97942733699124]
  We present MoMa-SG, a framework for building semantic-kinematic 3D scene graphs of articulated scenes.<n>We estimate articulation models using a novel unified twist estimation formulation.<n>We also introduce the novel Arti4D-Semantic dataset.
  arXiv  Detail & Related papers  (2026-02-18T10:40:35Z)
• CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining [4.039082584778385]
  We introduce Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining (CLAMP)<n>From the merged point cloud computed from RGB-D images and camera extrinsics, we re-render multi-view four-channel image observations with depth and 3D coordinates.<n>The pre-trained encoders learn to associate the 3D geometric and positional information of objects with robot action patterns.
  arXiv  Detail & Related papers  (2026-01-31T23:32:54Z)
• Object-centric 3D Motion Field for Robot Learning from Human Videos [56.9436352861611]
  We propose to use object-centric 3D motion field to represent actions for robot learning from human videos.<n>We present a novel framework for extracting this representation from videos for zero-shot control.<n> Experiments show that our method reduces 3D motion estimation error by over 50% compared to the latest method.
  arXiv  Detail & Related papers  (2025-06-04T17:59:06Z)
• Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction [51.49400490437258]
  This work develops a method for imitating articulated object manipulation from a single monocular RGB human demonstration. We first propose 4D Differentiable Part Models (4D-DPM), a method for recovering 3D part motion from a monocular video. Given this 4D reconstruction, the robot replicates object trajectories by planning bimanual arm motions that induce the demonstrated object part motion. We evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part trajectories and RSRD's physical execution performance on 9 objects across 10 trials each on a bimanual YuMi robot.
  arXiv  Detail & Related papers  (2024-09-26T17:57:16Z)
• Spot-Compose: A Framework for Open-Vocabulary Object Retrieval and Drawer Manipulation in Point Clouds [45.87961177297602]
  This work aims to integrate recent methods into a comprehensive framework for robotic interaction and manipulation in human-centric environments. Specifically, we leverage 3D reconstructions from a commodity 3D scanner for open-vocabulary instance segmentation. We show the performance and robustness of our model in two sets of real-world experiments including dynamic object retrieval and drawer opening.
  arXiv  Detail & Related papers  (2024-04-18T18:01:15Z)
• HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation [29.01984677695523]
  We introduce Hybrid Actor-Critic Maps for Manipulation (HACMan), a reinforcement learning approach for 6D non-prehensile manipulation of objects. We evaluate HACMan on a 6D object pose alignment task in both simulation and in the real world. Compared to alternative action representations, HACMan achieves a success rate more than three times higher than the best baseline.
  arXiv  Detail & Related papers  (2023-05-06T05:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.