LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World

• URL: http://arxiv.org/abs/2606.01458v1
• Date: Sun, 31 May 2026 21:36:02 GMT
• Title: LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World
• Abstract summary: LEGS (Loco-manipulation via Embodied Gaussian Splatting) is a hybrid simulator that composites a mesh foreground (robot, objects, props) over a photorealistic 3D Gaussian Splatting background reconstructed from a handheld scene capture.<n>On a Unitree G1 humanoid robot, across three pick-and-place tasks, a policy trained purely on LEGS data matches or exceeds one trained on human teleoperation demos on every experiment.<n>It also outperforms a mesh-only simulation baseline that ablates the effect of the 3DGS background, showing that photorealistic rendering is a key enabler for synthetic data transfer.
• Score: 14.534355249709371
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training vision-language-action (VLA) policies for humanoid loco-manipulation is constrained by the high cost and complexity of collecting human teleoperation demonstrations. VLA policies fine-tuned in simulators have, until now, failed to transfer effectively in humanoid loco-manipulation tasks. We present LEGS (Loco-manipulation via Embodied Gaussian Splatting), a hybrid simulator that composites a mesh foreground (robot, objects, props) over a photorealistic 3D Gaussian Splatting (3DGS) background reconstructed from a handheld scene capture. LEGS uses a procedural motion-primitive generator to synthesize labeled demonstrations at scale without human teleoperation, and a deterministic two-stage color calibration to align the rendered 3DGS image to the robot's deployment camera. On a Unitree G1 humanoid robot, across three pick-and-place tasks of increasing whole-body difficulty and three VLA backbones (psi_0, pi_0.5, GR00T N1.6), a policy trained purely on LEGS data matches or exceeds one trained on human teleoperation demos on every experiment. It also outperforms a mesh-only simulation baseline that ablates the effect of the 3DGS background, showing that photorealistic rendering is a key enabler for synthetic data transfer. Humanoid motion is recorded independently of scene appearance in LEGS, allowing the same auto-generated demonstrations to be re-rendered under new backgrounds and object meshes--covering a new scene at more than 15x lower cost than teleoperation--to augment training data for robustness to scene variations. Under combined object-and-scene appearance shift, the policy trained on re-rendered LEGS-AUG data maintains task success while the baseline trained on teleoperation data fails entirely. Our project page is located at https://legsvla.github.io/.

Related papers

• Pre-training Visual Dexterity in Simulation [67.17477115017896]
  We introduce a pre-training framework for dexterous manipulation that uses data entirely collected in simulation.<n>With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective.<n>We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation.
  arXiv  Detail & Related papers  (2026-08-16T20:21:33Z)
• VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes [71.14444761145435]
  Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion.<n>No existing data source provides this complete at scale.<n>We address this bottleneck by generating vision-language-kinematics supervision synthetically in reconstructed scenes.
  arXiv  Detail & Related papers  (2026-06-29T17:59:55Z)
• Contrastive Action-Image Pre-training for Visuomotor Control [91.5630572541332]
  We introduce CAIP, a vision encoder that treats human hand poses from large-scale egocentric video as a proxy for end-effector actions.<n>Leveraging 32,041 hours of egocentric human video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art vision encoders.
  arXiv  Detail & Related papers  (2026-06-15T20:00:20Z)
• OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation [51.37436817948304]
  OASIS is a simulation-data-driven framework for humanoid loco-manipulation.<n>It reconstructs realistic object assets from real-world images using a 3D generative model.<n>Under zero-shot deployment, it achieves higher success rates on most tasks than that trained on real-robot teleoperation data.
  arXiv  Detail & Related papers  (2026-06-07T10:01:53Z)
• GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors [113.71148915419246]
  We present GRAIL, a digital generation pipeline that composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot.<n>GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction.<n>We retarget the recovered motions to a humanoid robot and train complementary task-general morphology trackers.<n>GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting
  arXiv  Detail & Related papers  (2026-06-03T17:57:45Z)
• EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents [85.77432303199176]
  We propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones.<n>Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes.<n>Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via
  arXiv  Detail & Related papers  (2026-02-26T16:53:41Z)
• RoboPaint: From Human Demonstration to Any Robot and Any View [9.083647729839688]
  We propose a Real-Sim-Real data collection and data editing pipeline to transform human demonstrations into robot-executable, environment-specific training data.<n>We show that retargeted dex-hand trajectories achieve an 84% success rate across 10 diverse object manipulation tasks.<n>We offer a scalable, cost-effective alternative to teleoperation with minimal performance loss for complex dexterous manipulation.
  arXiv  Detail & Related papers  (2026-02-05T05:45:12Z)
• AHA! Animating Human Avatars in Diverse Scenes with Gaussian Splatting [26.560838721184435]
  We present a novel framework for animating humans in 3D scenes using 3D Gaussian Splatting (3DGS)<n>By representing humans and scenes as Gaussians, our approach allows for geometry-consistent free-viewpoint rendering of humans interacting with 3D scenes.<n>We evaluate our approach on scenes from Scannet++ and the SuperSplat library, and on avatars reconstructed from sparse and dense multi-view human capture.
  arXiv  Detail & Related papers  (2025-11-13T00:19:18Z)
• GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation [18.684526752120412]
  GSWorld is a photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines.<n>Our framework advocates "closing the loop" of developing manipulation policies with reproducible evaluation of policies learned from real-robot data.
  arXiv  Detail & Related papers  (2025-10-23T17:59:26Z)
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels [78.05453137978132]
  We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in within vectorized physics simulators.<n>This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPU.<n>We additionally demonstrate its applicability in a sim-to-real robotics setting.
  arXiv  Detail & Related papers  (2025-10-17T06:34:52Z)
• Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation [87.91642226587294]
  Current learning-based 3D reconstruction methods rely on the availability of captured real-world multi-view data.<n>We propose a self-distillation framework that distills the implicit 3D knowledge in the video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation.<n>Our framework achieves state-of-the-art performance in static and dynamic 3D scene generation.
  arXiv  Detail & Related papers  (2025-09-23T17:58:01Z)
• TrajBooster: Boosting Humanoid Whole-Body Manipulation via Trajectory-Centric Learning [79.59753528758361]
  We present TrajBooster, a cross-embodiment framework that leverages abundant wheeled-humanoid data to boost bipedal VLA.<n>Our key idea is to use end-effector trajectories as a morphology-agnostic interface.<n>Results show that TrajBooster allows existing wheeled-humanoid data to efficiently strengthen bipedal humanoid VLA performance.
  arXiv  Detail & Related papers  (2025-09-15T12:25:39Z)
• Robot Learning with Super-Linear Scaling [13.053949644385932]
  CASHER is a pipeline for scaling up data collection and learning in simulation where the performance scales superlinearly with human effort.<n>We show that CASHER enables fine-tuning of pre-trained policies to a target scenario using a video scan without any additional human effort.
  arXiv  Detail & Related papers  (2024-12-02T18:12:02Z)
• DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation [66.7719069053058]
  DeformGS is an approach to recover scene flow in highly deformable scenes using simultaneous video captures of a dynamic scene from multiple cameras. DeformGS improves 3D tracking by an average of 55.8% compared to the state-of-the-art. With sufficient texture, DeformGS achieves a median tracking error of 3.3 mm on a cloth of 1.5 x 1.5 m in area.
  arXiv  Detail & Related papers  (2023-11-30T18:53:03Z)
• Learning Motion Priors for 4D Human Body Capture in 3D Scenes [81.54377747405812]
  We propose LEMO: LEarning human MOtion priors for 4D human body capture. We introduce a novel motion prior, which reduces the jitters exhibited by poses recovered over a sequence. We also design a contact friction term and a contact-aware motion infiller obtained via per-instance self-supervised training. With our pipeline, we demonstrate high-quality 4D human body capture, reconstructing smooth motions and physically plausible body-scene interactions.
  arXiv  Detail & Related papers  (2021-08-23T20:47:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.