Real2Sim2Real for Vision-Language-Action Manipulation: An AMD ROCm-Based Pipeline
- URL: http://arxiv.org/abs/2607.22997v1
- Date: Sat, 25 Jul 2026 02:23:34 GMT
- Title: Real2Sim2Real for Vision-Language-Action Manipulation: An AMD ROCm-Based Pipeline
- Abstract summary: Physical AI is the integration of large vision--action models with embodied agents that act in the real world.<n>This paper presents an end-to-end, fully AMDaccelerated technology stack for embodied manipulation.
- Score: 13.511902327061856
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Physical AI -- the integration of large vision-language-action (VLA) models with embodied agents that act in the real world -- has emerged as the next major frontier for AI, echoed by industry leaders such as Jensen Huang (``the next big thing is Physical AI, AI with a body,'' GTC Paris, June 2025) and Dr. Lisa Su (`we're entering the world of Physical AI ... this is where AI enters the real world,' CES 2026). This paper presents an end-to-end, fully AMD-accelerated technology stack for embodied manipulation, spanning data-center training silicon, Radeon PRO simulation/rendering GPUs, and Ryzen AI edge compute, unified by the open ROCm software stack. We demonstrate that training and deploying VLA-based manipulation policies does not require a CUDA-locked ecosystem. Four progressive demonstrations are presented: (1) a Sim-to-Real manipulation pipeline trained with SmolVLA and deployed on a physical Franka arm; (2) a semantic, language-grounded object-selection task (`one-of-three'); (3) a Real2Sim synthetic-data generation pipeline that fuses 3D Gaussian Splatting (3DGS) reconstructions of real scenes with the Genesis physics engine; and (4) large-scale reinforcement learning for quadruped and humanoid locomotion benchmarked across multiple hardware platforms. All pipelines run natively on ROCm + PyTorch on RDNA4 (Radeon AI PRO R9700) and RDNA3.5 (Radeon PRO W7900) hardware and are reproducible on the free Radeon Cloud Platform.
Related papers
- Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation [88.94027963091206]
We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence.<n>It predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer.
arXiv Detail & Related papers (2026-06-15T17:52:31Z) - Cosmos 3: Omnimodal World Models for Physical AI [321.3056180868173]
We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences.<n>Our evaluation demonstrates that Cosmos 3 unifies a new state-of-the-art across a diverse suite of understanding and generation tasks.
arXiv Detail & Related papers (2026-06-01T19:12:30Z) - GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation [54.671815855499034]
Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors to action generation.<n>Standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution.<n>We propose textbfGaussianDream, a feed-forward 3D Gaussian world-model plug-in.
arXiv Detail & Related papers (2026-05-20T05:51:30Z) - V-Dreamer: Automating Robotic Simulation and Trajectory Synthesis via Video Generation Priors [30.872127145725187]
V-Dreamer is a fully automated framework that generates open-vocabulary, simulation-ready manipulation environments.<n>V-Dreamer employs a novel generative pipeline that constructs physically grounded 3D scenes.<n>We show that our policies robustly generalize to unseen objects in simulation and achieve effective sim-to-real transfer.
arXiv Detail & Related papers (2026-03-19T12:00:04Z) - World Simulation with Video Foundation Models for Physical AI [181.78265691303315]
We release [Cosmos-Predict2.5] and [Cosmos-Transfer2.5] as versatile tools for scaling embodied intelligence.<n>We release source code, pretrained checkpoints, and curated benchmarks under the NVIDIA Open Model License.
arXiv Detail & Related papers (2025-10-28T22:44:13Z) - GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation [18.684526752120412]
GSWorld is a photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines.<n>Our framework advocates "closing the loop" of developing manipulation policies with reproducible evaluation of policies learned from real-robot data.
arXiv Detail & Related papers (2025-10-23T17:59:26Z) - GaussGym: An open-source real-to-sim framework for learning locomotion from pixels [78.05453137978132]
We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in within vectorized physics simulators.<n>This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPU.<n>We additionally demonstrate its applicability in a sim-to-real robotics setting.
arXiv Detail & Related papers (2025-10-17T06:34:52Z) - GWM: Towards Scalable Gaussian World Models for Robotic Manipulation [53.51622803589185]
We propose a novel branch of world model named Gaussian World Model (GWM) for robotic manipulation.<n>At its core is a latent Diffusion Transformer (DiT) combined with a 3D variational autoencoder, enabling fine-grained scene-level future state reconstruction.<n>Both simulated and real-world experiments depict that GWM can precisely predict future scenes conditioned on diverse robot actions.
arXiv Detail & Related papers (2025-08-25T02:01:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.