A Multifidelity Sim-to-Real Pipeline for Verifiable and Compositional Reinforcement Learning

• URL: http://arxiv.org/abs/2312.01249v1
• Date: Sat, 2 Dec 2023 23:46:27 GMT
• Title: A Multifidelity Sim-to-Real Pipeline for Verifiable and Compositional Reinforcement Learning
• Abstract summary: We propose and demonstrate a compositional framework for training and verifying reinforcement learning systems. By decomposing complex robotic tasks into component subtasks, the framework allows for the independent training and testing of the corresponding subtask policies. In an experimental case study we apply the framework to train and deploy a compositional RL system that successfully pilots a Warthog unmanned ground robot.
• Score: 18.42455580047624
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose and demonstrate a compositional framework for training and verifying reinforcement learning (RL) systems within a multifidelity sim-to-real pipeline, in order to deploy reliable and adaptable RL policies on physical hardware. By decomposing complex robotic tasks into component subtasks and defining mathematical interfaces between them, the framework allows for the independent training and testing of the corresponding subtask policies, while simultaneously providing guarantees on the overall behavior that results from their composition. By verifying the performance of these subtask policies using a multifidelity simulation pipeline, the framework not only allows for efficient RL training, but also for a refinement of the subtasks and their interfaces in response to challenges arising from discrepancies between simulation and reality. In an experimental case study we apply the framework to train and deploy a compositional RL system that successfully pilots a Warthog unmanned ground robot.

Related papers

• Small Model as Master Orchestrator: Learning Unified Agent-Tool Orchestration with Parallel Subtask Decomposition [61.291733522717415]
  Agent-as-Tool is a unified parallel orchestration paradigm that mitigates both agents and tools into a standardized, learnable action space.<n>ParaManager decouples planning decisions from subtask solving, enabling state-aware parallel subtask decomposition, delegation, and asynchronous execution.<n>Experiments show that ParaManager achieves strong performance across multiple benchmarks and exhibits robust generalization under unseen model pools.
  arXiv  Detail & Related papers  (2026-04-18T14:41:27Z)
• E2E-Fly: An Integrated Training-to-Deployment System for End-to-End Quadrotor Autonomy [12.44680847653453]
  E2E-Fly is an integrated framework featuring an agile quadrotor platform and a full-stack training, validation, and deployment workflow.<n>We introduce a two-stage validation strategy using sim-to-sim transfer and hardware-in-the-loop testing, and deploy policies onto two physical quadrotor platforms.<n>We demonstrate the effectiveness of our framework for training six end-to-end control tasks and deploy them in the real world.
  arXiv  Detail & Related papers  (2026-04-14T16:00:22Z)
• ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenas [13.919124676472022]
  ASTRA is an end-to-end framework for training tool-augmented language model agents.<n>ASTRA integrates scalable data synthesis and verifiable reinforcement learning.<n> Experiments on multiple agentic tool-use benchmarks demonstrate that ASTRA-trained models achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-01-29T11:22:23Z)
• Scaling Agent Learning via Experience Synthesis [100.42712232390532]
  Reinforcement learning can empower autonomous agents by enabling self-improvement through interaction.<n>But its practical adoption remains challenging due to costly rollouts, limited task diversity, unreliable reward signals, and infrastructure complexity.<n>We introduce DreamGym, the first unified framework designed to synthesize diverse experiences with scalability in mind.
  arXiv  Detail & Related papers  (2025-11-05T18:58:48Z)
• Simulating Environments with Reasoning Models for Agent Training [55.98861707136674]
  Building bespoke environments for training is heavy, brittle, and limits progress.<n>We propose two frameworks: Simia-SFT and Simia-RL.<n>Simia-SFT and Simia-RL enable scalable agent training without environment engineering.
  arXiv  Detail & Related papers  (2025-11-03T18:29:57Z)
• MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering [57.156093929365255]
  Gym-style framework for systematically reinforcement learning, evaluating, and improving autonomous large language model (LLM) agents.<n>MLE-Dojo covers diverse, open-ended MLE tasks carefully curated to reflect realistic engineering scenarios.<n>Its fully executable environment supports comprehensive agent training via both supervised fine-tuning and reinforcement learning.
  arXiv  Detail & Related papers  (2025-05-12T17:35:43Z)
• A General Infrastructure and Workflow for Quadrotor Deep Reinforcement Learning and Reality Deployment [48.90852123901697]
  We propose a platform that enables seamless transfer of end-to-end deep reinforcement learning (DRL) policies to quadrotors. Our platform provides rich types of environments including hovering, dynamic obstacle avoidance, trajectory tracking, balloon hitting, and planning in unknown environments.
  arXiv  Detail & Related papers  (2025-04-21T14:25:23Z)
• Multi-Task Reinforcement Learning for Quadrotors [18.71563817810032]
  This paper presents a novel multi-task reinforcement learning (MTRL) framework tailored for quadrotor control. By employing a multi-critic architecture and shared task encoders, our framework facilitates knowledge transfer across tasks, enabling a single policy to execute diverse maneuvers.
  arXiv  Detail & Related papers  (2024-12-17T01:10:18Z)
• Guiding Multi-agent Multi-task Reinforcement Learning by a Hierarchical Framework with Logical Reward Shaping [16.5526277899717]
  This study aims to design a multi-agent cooperative algorithm with logic reward shaping. Experiments have been conducted on various types of tasks in the Minecraft-like environment.
  arXiv  Detail & Related papers  (2024-11-02T09:03:23Z)
• Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning [53.3760591018817]
  We propose a new benchmarking environment for aquatic navigation using recent advances in the integration between game engines and Deep Reinforcement Learning. Specifically, we focus on PPO, one of the most widely accepted algorithms, and we propose advanced training techniques. Our empirical evaluation shows that a well-designed combination of these ingredients can achieve promising results.
  arXiv  Detail & Related papers  (2024-05-30T23:20:23Z)
• Mixed-Reality Digital Twins: Leveraging the Physical and Virtual Worlds for Hybrid Sim2Real Transition of Multi-Agent Reinforcement Learning Policies [0.0]
  Multi-agent reinforcement learning (MARL) for cyber-physical vehicle systems usually requires a significantly long training time. This work presents a mixed-reality digital twin framework capable of selectively scaling parallelized workloads on-demand. We study the effect of: (i) agent and environment parallelization on training time, and (ii) systematic domain randomization on zero-shot sim2real transfer.
  arXiv  Detail & Related papers  (2024-03-16T18:47:04Z)
• Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents [9.529492371336286]
  Reinforcement Learning (RL) has made significant strides in enabling artificial agents to learn diverse behaviors. We propose a novel approach, called Logical Specifications-guided Dynamic Task Sampling (LSTS) LSTS learns a set of RL policies to guide an agent from an initial state to a goal state based on a high-level task specification.
  arXiv  Detail & Related papers  (2024-02-06T04:00:21Z)
• Resilient Control of Networked Microgrids using Vertical Federated Reinforcement Learning: Designs and Real-Time Test-Bed Validations [5.394255369988441]
  This paper presents a novel federated reinforcement learning (Fed-RL) approach to tackle (a) model complexities, unknown dynamical behaviors of IBR devices, (b) privacy issues regarding data sharing in multi-party-owned networked grids, and (2) transfers learned controls from simulation to hardware-in-the-loop test-bed. Experiments show that the simulator-trained RL controllers produce convincing results with the real-time test-bed set-up, validating the minimization of sim-to-real gap.
  arXiv  Detail & Related papers  (2023-11-21T00:59:27Z)
• Learning to Sail Dynamic Networks: The MARLIN Reinforcement Learning Framework for Congestion Control in Tactical Environments [53.08686495706487]
  This paper proposes an RL framework that leverages an accurate and parallelizable emulation environment to reenact the conditions of a tactical network. We evaluate our RL learning framework by training a MARLIN agent in conditions replicating a bottleneck link transition between a Satellite Communication (SATCOM) and an UHF Wide Band (UHF) radio link.
  arXiv  Detail & Related papers  (2023-06-27T16:15:15Z)
• A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving [0.0]
  Deep Reinforcement Learning (DRL) has shown remarkable success in solving complex tasks. transferring DRL agents to the real world is still challenging due to the significant discrepancies between simulation and reality. We propose a robust DRL framework that leverages platform-dependent perception modules to extract task-relevant information.
  arXiv  Detail & Related papers  (2023-04-14T07:55:07Z)
• Multitask Adaptation by Retrospective Exploration with Learned World Models [77.34726150561087]
  We propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from task-agnostic storage. The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage.
  arXiv  Detail & Related papers  (2021-10-25T20:02:57Z)
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments [89.04823188871906]
  Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. Most of the existing simulators rely on randomly generating the environments. We introduce the benefits of adopting an existing formal scenario specification language, SCENIC, to assist researchers.
  arXiv  Detail & Related papers  (2021-06-18T21:49:46Z)
• Composable Learning with Sparse Kernel Representations [110.19179439773578]
  We present a reinforcement learning algorithm for learning sparse non-parametric controllers in a Reproducing Kernel Hilbert Space. We improve the sample complexity of this approach by imposing a structure of the state-action function through a normalized advantage function. We demonstrate the performance of this algorithm on learning obstacle-avoidance policies in multiple simulations of a robot equipped with a laser scanner while navigating in a 2D environment.
  arXiv  Detail & Related papers  (2021-03-26T13:58:23Z)
• Learning Centric Wireless Resource Allocation for Edge Computing: Algorithm and Experiment [15.577056429740951]
  Edge intelligence is an emerging network architecture that integrates sensing, communication, computing components, and supports various machine learning applications. Existing methods ignore two important facts: 1) different models have heterogeneous demands on training data; 2) there is a mismatch between the simulated environment and the real-world environment. This paper proposes the learning centric wireless resource allocation scheme that maximizes the worst learning performance of multiple tasks.
  arXiv  Detail & Related papers  (2020-10-29T06:20:40Z)
• RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real [74.45688231140689]
  We introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. We obtain RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning.
  arXiv  Detail & Related papers  (2020-06-16T08:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.