SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments

• URL: http://arxiv.org/abs/2504.21454v1
• Date: Wed, 30 Apr 2025 09:22:55 GMT
• Title: SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments
• Authors: Federico Nesti, Gianluca D'Amico, Mauro Marinoni, Giorgio Buttazzo,
• Abstract summary: This paper presents SimPRIVE, a simulation framework for physical robot interaction with virtual environments.<n>Using SimPRIVE, any physical mobile robot running on ROS 2 can easily be configured to move its digital twin in a virtual world built with the Unreal Engine 5 graphic engine.<n>The framework has been validated by testing a reinforcement learning agent trained for obstacle avoidance on an AgileX Scout Mini rover.
• Score: 4.966661313606916
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The use of machine learning in cyber-physical systems has attracted the interest of both industry and academia. However, no general solution has yet been found against the unpredictable behavior of neural networks and reinforcement learning agents. Nevertheless, the improvements of photo-realistic simulators have paved the way towards extensive testing of complex algorithms in different virtual scenarios, which would be expensive and dangerous to implement in the real world. This paper presents SimPRIVE, a simulation framework for physical robot interaction with virtual environments, which operates as a vehicle-in-the-loop platform, rendering a virtual world while operating the vehicle in the real world. Using SimPRIVE, any physical mobile robot running on ROS 2 can easily be configured to move its digital twin in a virtual world built with the Unreal Engine 5 graphic engine, which can be populated with objects, people, or other vehicles with programmable behavior. SimPRIVE has been designed to accommodate custom or pre-built virtual worlds while being light-weight to contain execution times and allow fast rendering. Its main advantage lies in the possibility of testing complex algorithms on the full software and hardware stack while minimizing the risks and costs of a test campaign. The framework has been validated by testing a reinforcement learning agent trained for obstacle avoidance on an AgileX Scout Mini rover that navigates a virtual office environment where everyday objects and people are placed as obstacles. The physical rover moves with no collision in an indoor limited space, thanks to a LiDAR-based heuristic.

Related papers

• Unreal Robotics Lab: A High-Fidelity Robotics Simulator with Advanced Physics and Rendering [4.760567755149477]
  This paper presents a novel simulation framework that integrates the Unreal Engine's advanced rendering capabilities with MuJoCo's high-precision physics simulation.<n>Our approach enables realistic robotic perception while maintaining accurate physical interactions.<n>We benchmark visual navigation and SLAM methods within our framework, demonstrating its utility for testing real-world robustness in controlled yet diverse scenarios.
  arXiv  Detail & Related papers  (2025-04-19T01:54:45Z)
• VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion [25.440573256776133]
  This paper presents a Real-to-Sim-to-Real framework that generates and physically interactive "digital twin" simulation environments for visual navigation and locomotion learning.
  arXiv  Detail & Related papers  (2025-02-03T17:15:05Z)
• Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation [62.5805866419814]
  Vid2Sim is a novel framework that bridges the sim2real gap through a scalable and cost-efficient real2sim pipeline for neural 3D scene reconstruction and simulation.<n>Experiments demonstrate that Vid2Sim significantly improves the performance of urban navigation in the digital twins and real world by 31.2% and 68.3% in success rate.
  arXiv  Detail & Related papers  (2025-01-12T03:01:15Z)
• DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation [54.02069690134526]
  We propose DrivingSphere, a realistic and closed-loop simulation framework. Its core idea is to build 4D world representation and generate real-life and controllable driving scenarios. By providing a dynamic and realistic simulation environment, DrivingSphere enables comprehensive testing and validation of autonomous driving algorithms.
  arXiv  Detail & Related papers  (2024-11-18T03:00:33Z)
• Learning to navigate efficiently and precisely in real environments [14.52507964172957]
  Embodied AI literature focuses on end-to-end agents trained in simulators like Habitat or AI-Thor. In this work we explore end-to-end training of agents in simulation in settings which minimize the sim2real gap.
  arXiv  Detail & Related papers  (2024-01-25T17:50:05Z)
• How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence [16.24370001383615]
  How to adapt driving knowledge learned in simulation to reality becomes a critical issue. Virtual simulation world differs from the real world in many aspects such as lighting, textures, vehicle dynamics, and agents' behaviors. Three categories of approaches to address the reality gap issue: transferring knowledge from simulation to reality (sim2real), learning in digital twins (DTs), and learning by parallel intelligence (PI)
  arXiv  Detail & Related papers  (2023-05-02T09:00:32Z)
• Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data [56.49494318285391]
  We present Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world.
  arXiv  Detail & Related papers  (2022-10-25T17:50:36Z)
• DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality [64.51295032956118]
  We train a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand. Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups.
  arXiv  Detail & Related papers  (2022-10-25T01:51:36Z)
• Real2Sim2Real Transfer for Control of Cable-driven Robots via a Differentiable Physics Engine [9.268539775233346]
  Tensegrity robots exhibit high strength-to-weight ratios and significant deformations. They are hard to control, however, due to high dimensionality, complex dynamics, and a coupled architecture. This paper describes a Real2Sim2Real (R2S2R) strategy for modeling tensegrity robots.
  arXiv  Detail & Related papers  (2022-09-13T18:51:26Z)
• Data generation using simulation technology to improve perception mechanism of autonomous vehicles [0.0]
  We will demonstrate the effectiveness of combining data gathered from the real world with data generated in the simulated world to train perception systems. We will also propose a multi-level deep learning perception framework that aims to emulate a human learning experience.
  arXiv  Detail & Related papers  (2022-07-01T03:42:33Z)
• VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles [131.2240621036954]
  We present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras. We demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle.
  arXiv  Detail & Related papers  (2021-11-23T18:58:10Z)
• LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World [84.57894492587053]
  We develop a novel simulator that captures both the power of physics-based and learning-based simulation. We first utilize ray casting over the 3D scene and then use a deep neural network to produce deviations from the physics-based simulation. We showcase LiDARsim's usefulness for perception algorithms-testing on long-tail events and end-to-end closed-loop evaluation on safety-critical scenarios.
  arXiv  Detail & Related papers  (2020-06-16T17:44:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.