Grounding Sim-to-Real Generalization in Dexterous Manipulation: An Empirical Study with Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.22876v1
• Date: Tue, 24 Mar 2026 07:21:29 GMT
• Title: Grounding Sim-to-Real Generalization in Dexterous Manipulation: An Empirical Study with Vision-Language-Action Models
• Abstract summary: Given the cost of real-world data collection, a practical alternative is to generate synthetic data through simulation.<n>We empirically examine the primary determinants of Sim-to-Real generalization across four dimensions.<n>We derive critical insights into Sim-to-Real transfer through experiments involving over 10k real-world trials.
• Score: 26.206956451239105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning a generalist control policy for dexterous manipulation typically relies on large-scale datasets. Given the high cost of real-world data collection, a practical alternative is to generate synthetic data through simulation. However, the resulting synthetic data often exhibits a significant gap from real-world distributions. While many prior studies have proposed algorithms to bridge the Sim-to-Real discrepancy, there remains a lack of principled research that grounds these methods in real-world manipulation tasks, particularly their performance on generalist policies such as Vision-Language-Action (VLA) models. In this study, we empirically examine the primary determinants of Sim-to-Real generalization across four dimensions: multi-level domain randomization, photorealistic rendering, physics-realistic modeling, and reinforcement learning updates. To support this study, we design a comprehensive evaluation protocol to quantify the real-world performance of manipulation tasks. The protocol accounts for key variations in background, lighting, distractors, object types, and spatial features. Through experiments involving over 10k real-world trials, we derive critical insights into Sim-to-Real transfer. To inform and advance future studies, we release both the robotic platforms and the evaluation protocol for public access to facilitate independent verification, thereby establishing a realistic and standardized benchmark for dexterous manipulation policies.

Related papers

• RoboSynChallenge: Mastering Real-World Dexterity via Generalizing Synthesized Manipulation Skills [90.48218237655708]
  RoboSynChallenge aims to foster the development of broadly capable, data-efficient, and adaptable manipulation systems.<n>Challenge integrates large-scale synthetic data generation with standardized real-world robotic evaluation.
  arXiv  Detail & Related papers  (2026-08-12T04:29:00Z)
• PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies [88.78188489161028]
  We introduce Policy Evaluation and Environment Reconstruction in Simulation (PolaRiS)<n>PolaRiS is a scalable real-to-sim framework for high-fidelity simulated robot evaluation.<n>We show that PolaRiS evaluations provide a much stronger correlation to real world generalist policy performance than existing simulated benchmarks.
  arXiv  Detail & Related papers  (2025-12-18T18:49:41Z)
• SimScale: Learning to Drive via Real-World Simulation at Scale [45.08991279559151]
  We introduce a novel and scalable simulation framework capable of synthesizing massive unseen states upon existing driving logs.<n>Our pipeline utilizes advanced neural rendering with a reactive environment to generate high-fidelity multi-view observations.<n>We develop a pseudo-expert trajectory generation mechanism for these newly simulated states to provide action supervision.
  arXiv  Detail & Related papers  (2025-11-28T17:17:38Z)
• R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation [74.41728218960465]
  We propose a real-to-real 3D data generation framework (R2RGen) that directly augments the pointcloud observation-action pairs to generate real-world data.<n>R2RGen substantially enhances data efficiency on extensive experiments and demonstrates strong potential for scaling and application on mobile manipulation.
  arXiv  Detail & Related papers  (2025-10-09T17:55:44Z)
• Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training [21.855770200309674]
  We propose a unified sim-and-real co-training framework for learning generalizable manipulation policies.<n>We show it can leverage abundant simulation data to achieve up to a 30% improvement in the real-world success rate.
  arXiv  Detail & Related papers  (2025-09-23T04:32:53Z)
• PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrations [24.77819842428131]
  Reinforcement learning can autonomously explore to obtain robust behaviors.<n>Training RL agents through direct interaction with the real world is often impractical and unsafe.<n>We propose an integrated real-to-sim-to-real pipeline that constructs simulation environments based on expert demonstrations.
  arXiv  Detail & Related papers  (2025-04-29T08:01:27Z)
• Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids [56.892520712892804]
  We introduce a practical sim-to-real RL recipe that trains a humanoid robot to perform three dexterous manipulation tasks.<n>We demonstrate high success rates on unseen objects and robust, adaptive policy behaviors.
  arXiv  Detail & Related papers  (2025-02-27T18:59:52Z)
• ManiBox: Enhancing Spatial Grasping Generalization via Scalable Simulation Data Generation [37.73074657448699]
  bfManiBox is a novel bounding-box-guided manipulation method built on a simulation-based teacher-student framework.<n>ManiBox demonstrates a marked improvement in spatial grasping generalization and adaptability to diverse objects and backgrounds.
  arXiv  Detail & Related papers  (2024-11-04T07:05:02Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models.<n>Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning.<n>Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation [62.22058066456076]
  Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks. RL does not work directly in the real-world, which is known as the sim-to-real transfer problem. We propose a method that learns on an observation space constructed by point clouds and environment randomization.
  arXiv  Detail & Related papers  (2020-07-27T17:46:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.