LACE: Latent Visual Representation for Cross-Embodiment Learning

• URL: http://arxiv.org/abs/2605.16743v1
• Date: Sat, 16 May 2026 01:50:18 GMT
• Title: LACE: Latent Visual Representation for Cross-Embodiment Learning
• Abstract summary: Cross-embodiment learning from human demonstrations is hindered by the visual gap between human and robot embodiments.<n>We propose LACE, a framework that aligns human and robot visual representations in the latent space of self-supervised learning backbones.
• Score: 36.1587655731958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-embodiment learning from human demonstrations is hindered by the visual gap between human and robot embodiments. While self-supervised learning (SSL) backbones encode rich inter-class semantics of general objects, we show they fail to establish correspondence between human and robot hands. We propose LACE, a framework that aligns human and robot visual representations in the latent space of these backbones by leveraging correspondences between shared body parts across embodiments as sparse supervision. These annotations can be automatically obtained via forward kinematics, and single robot demonstration is sufficient to train the model. Our semantic alignment loss matches distributions incurred by corresponding features, lifting patch-level supervision to semantic-level alignment, while a Gram loss preserves pretrained feature quality. This alignment enables robot policies to leverage abundant human data when robot demonstrations are scarce: in zero-shot transfer, policies using LACE-DINO outperform those using DINO by a large margin (65\%), with consistent gains in low-data regimes and out-of-distribution environments.

Related papers

• AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning [25.241504281004243]
  We present AdvDex, a framework for learning dexterous manipulation from human and robot demonstrations.<n>First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations.<n>Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an $mathrmSE(3)$ wrist pose and 15 finger joints.<n>Third, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation.
  arXiv  Detail & Related papers  (2026-08-14T07:19:06Z)
• HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model [13.153130497047043]
  Learning generalizable vision-language-action models from large-scale human videos is promising but challenging due to cross-embodiment discrepancies.<n>We propose HARP, a human-robot aligned representation learning framework for more effective VLA pretraining from human videos.<n>Experiments on feature visualization, simulation, and realworld manipulation show improved human-robot alignment and downstream policy performance.
  arXiv  Detail & Related papers  (2026-05-29T12:36:30Z)
• Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing [35.3765036136913]
  We propose a generative framework for cross-embodiment video editing.<n>Our method factorizes a demonstration video into two latent spaces by enforcing a dual contrastive objective.<n>A parameter-efficient adapter injects these latent codes into a frozen video diffusion model, enabling the synthesis of a coherent robot execution video.
  arXiv  Detail & Related papers  (2026-05-05T11:09:41Z)
• UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modeling [41.44664297185315]
  We introduce UniT, a framework that establishes a unified physical language for human-to-humanoid transfer.<n>UniT employs a tri-branch cross-reconstruction mechanism: actions predict vision to anchor kinematics to physical outcomes, while vision reconstructs actions to filter out irrelevant visual confounders.
  arXiv  Detail & Related papers  (2026-04-21T17:57:27Z)
• LIDEA: Human-to-Robot Imitation Learning via Implicit Feature Distillation and Explicit Geometry Alignment [48.809726040947226]
  We introduce LIDEA, an imitation learning framework in which policy learning benefits from human demonstrations.<n>In the 2D visual domain, LIDEA employs a dual-stage transitive distillation pipeline that aligns human and robot representations in a shared latent space.<n>In the 3D geometric domain, we propose an embodiment-agnostic alignment strategy that explicitly decouples embodiment from interaction geometry.<n>Results show that human data substitutes up to 80% of costly robot demonstrations, and the framework successfully transfers unseen patterns from human videos for out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-04-12T15:02:34Z)
• Cross-Hand Latent Representation for Vision-Language-Action Models [49.32460749933983]
  Training reliable vision-language-action models for dexterous manipulation requires large-scale demonstrations across many robotic hands.<n>We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands.
  arXiv  Detail & Related papers  (2026-03-10T18:50:57Z)
• Learning a Unified Latent Space for Cross-Embodiment Robot Control [8.76482934005711]
  We present a scalable framework for cross-embodiment humanoid robot control.<n>We learn a shared latent representation that unifies motion across humans and diverse humanoid platforms.<n>Our method supports the efficient addition of new robots to the latent space.
  arXiv  Detail & Related papers  (2026-01-21T19:31:17Z)
• MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training [102.850162490626]
  We propose MiVLA, a vision-language-action model empowered by human-robot mutual imitation pre-training.<n>We show that MiVLA achieves strong improved generalization capability, outperforming state-of-the-art VLAs.
  arXiv  Detail & Related papers  (2025-12-17T12:59:41Z)
• OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction [76.44108003274955]
  A dominant paradigm for teaching humanoid robots complex skills is to retarget human motions as kinematic references to train reinforcement learning policies.<n>We introduce OmniRetarget, an interaction-preserving data generation engine based on an interaction mesh.<n>By minimizing the Laplacian deformation between the human and robot meshes, OmniRetarget generates kinematically feasible trajectories.
  arXiv  Detail & Related papers  (2025-09-30T17:59:02Z)
• One-Shot Imitation under Mismatched Execution [7.060120660671016]
  Human demonstrations are a powerful way to program robots to do long-horizon manipulation tasks.<n> translating these demonstrations into robot-executable actions presents significant challenges due to execution mismatches in movement styles and physical capabilities.<n>We propose RHyME, a novel framework that automatically pairs human and robot trajectories using sequence-level optimal transport cost functions.
  arXiv  Detail & Related papers  (2024-09-10T16:11:57Z)
• Visual Affordance Prediction for Guiding Robot Exploration [56.17795036091848]
  We develop an approach for learning visual affordances for guiding robot exploration. We use a Transformer-based model to learn a conditional distribution in the latent embedding space of a VQ-VAE. We show how the trained affordance model can be used for guiding exploration by acting as a goal-sampling distribution, during visual goal-conditioned policy learning in robotic manipulation.
  arXiv  Detail & Related papers  (2023-05-28T17:53:09Z)
• MILD: Multimodal Interactive Latent Dynamics for Learning Human-Robot Interaction [34.978017200500005]
  We propose Multimodal Interactive Latent Dynamics (MILD) to address the problem of two-party physical Human-Robot Interactions (HRIs) We learn the interaction dynamics from demonstrations, using Hidden Semi-Markov Models (HSMMs) to model the joint distribution of the interacting agents in the latent space of a Variational Autoencoder (VAE) MILD generates more accurate trajectories for the controlled agent (robot) when conditioned on the observed agent's (human) trajectory.
  arXiv  Detail & Related papers  (2022-10-22T11:25:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.