Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning

• URL: http://arxiv.org/abs/2603.09427v2
• Date: Thu, 12 Mar 2026 14:26:08 GMT
• Title: Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning
• Abstract summary: This work systematically analyzes how core Markov Decision Process (MDP) design choices affect this transfer.<n>Using a color mixing task, we evaluate different MDP configurations and mixing dynamics across simulation and real-world experiments.<n>We validate our findings on physical hardware, demonstrating that physics-based dynamics models achieve up to 50% real-world success.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement Learning (RL) has demonstrated strong potential for industrial process control, yet policies trained in simulation often suffer from a significant sim-to-real gap when deployed on physical hardware. This work systematically analyzes how core Markov Decision Process (MDP) design choices -- state composition, target inclusion, reward formulation, termination criteria, and environment dynamics models -- affect this transfer. Using a color mixing task, we evaluate different MDP configurations and mixing dynamics across simulation and real-world experiments. We validate our findings on physical hardware, demonstrating that physics-based dynamics models achieve up to 50% real-world success under strict precision constraints where simplified models fail entirely. Our results provide practical MDP design guidelines for deploying RL in industrial process control.

Related papers

• Generalized Task-Driven Design of Soft Robots via Reduced-Order FEM-based Surrogate Modeling [8.36237361666539]
  This paper presents a unified reduced-order finite element method (FEM)-based surrogate modeling pipeline for task-driven soft robot design.<n>High-fidelity FEM simulations characterize actuator behavior at the modular level, from which compact surrogate joint models are constructed.<n>A meta-model maps actuator design parameters to surrogate representations, enabling rapid instantiation across a parameterized actuator family.
  arXiv  Detail & Related papers  (2026-03-20T09:33:09Z)
• Simulation-based Learning of Electrical Cabinet Assembly Using Robot Skills [2.900269335003268]
  This paper presents a simulation-driven approach for automating the force-controlled assembly of electrical terminals on DIN-rails.<n>The proposed method integrates deep reinforcement learning (DRL) with parameterizable robot skills in a physics-based simulation environment.<n> Experimental results demonstrate high success rates (up to 100%) in both simulation and real-world settings, even under significant positional and rotational deviations.
  arXiv  Detail & Related papers  (2026-02-16T08:45:54Z)
• Reshaping Action Error Distributions for Reliable Vision-Language-Action Models [69.38615670891038]
  In robotic manipulation, vision-language-action (VLA) models have emerged as a promising paradigm for learning generalizable and scalable robot policies.<n>We focus on continuous-action VLA models and move beyond conventional MSE-based regression by reshaping action error distributions during training.<n>We evaluate our approaches across standard, few-shot, and noisy settings on multiple representative VLA architectures.
  arXiv  Detail & Related papers  (2026-02-04T05:37:09Z)
• Aligning Agentic World Models via Knowledgeable Experience Learning [68.85843641222186]
  We introduce WorldMind, a framework that constructs a symbolic World Knowledge Repository by synthesizing environmental feedback.<n>WorldMind achieves superior performance compared to baselines with remarkable cross-model and cross-environment transferability.
  arXiv  Detail & Related papers  (2026-01-19T17:33:31Z)
• Do-Undo: Generating and Reversing Physical Actions in Vision-Language Models [57.71440995598757]
  We introduce the Do-Undo task and benchmark to address a critical gap in vision-language models.<n>Do-Undo requires models to simulate the outcome of a physical action and then accurately reverse it, reflecting true cause-and-effect in the visual world.
  arXiv  Detail & Related papers  (2025-12-15T18:03:42Z)
• Analyzing Planner Design Trade-offs for MAPF under Realistic Simulation [8.088161779831582]
  We study how planner design choices influence performance under realistic execution settings.<n>We highlight open challenges and research directions to steer the community toward practical, real-world deployment.
  arXiv  Detail & Related papers  (2025-12-10T15:15:26Z)
• LLMs for Analog Circuit Design Continuum (ACDC) [10.15990970510281]
  Large Language Models (LLMs) and transformer architectures have shown impressive reasoning and generation capabilities.<n>This work focuses on AI-assisted design where humans remain in the loop.<n>Our results highlight key reliability challenges, including sensitivity to data format, instability in generated designs, and limited generalization to unseen circuit configurations.
  arXiv  Detail & Related papers  (2025-12-09T23:57:28Z)
• SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models [60.80050275581661]
  Vision-Language Models (VLMs) exhibit remarkable common-sense and semantic reasoning capabilities.<n>They lack a grounded understanding of physical dynamics.<n>We present S, a test-time, SIMulation-enabled ACTion Planning framework.<n>Our method demonstrates state-of-the-art performance on five challenging, real-world rigid-body and deformable manipulation tasks.
  arXiv  Detail & Related papers  (2025-12-05T18:51:03Z)
• Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling [31.74241286023207]
  Reinforcement learning (RL) is playing an increasingly important role in fields such as robotic control and autonomous driving.<n>The gap between simulation and the real environment remains a major obstacle to the practical deployment of RL.<n>We propose a latent space based approach to analyze the impact of simulation on real-world policy improvement.
  arXiv  Detail & Related papers  (2025-06-15T06:02:42Z)
• Re4MPC: Reactive Nonlinear MPC for Multi-model Motion Planning via Deep Reinforcement Learning [0.94371657253557]
  We propose a novel multi-model motion planning pipeline termed Re4MPC.<n>It computes trajectories in reaching end-effector goals using Model Predictive Control.<n>We show that Re4MPC is more computationally efficient and achieves higher success rates than the NMPC baseline.
  arXiv  Detail & Related papers  (2025-06-10T01:58:32Z)
• GausSim: Foreseeing Reality by Gaussian Simulator for Elastic Objects [55.02281855589641]
  GausSim is a novel neural network-based simulator designed to capture the dynamic behaviors of real-world elastic objects represented through Gaussian kernels.<n>We leverage continuum mechanics and treat each kernel as a Center of Mass System (CMS) that represents continuous piece of matter.<n>In addition, GausSim incorporates explicit physics constraints, such as mass and momentum conservation, ensuring interpretable results and robust, physically plausible simulations.
  arXiv  Detail & Related papers  (2024-12-23T18:58:17Z)
• HarmonyDream: Task Harmonization Inside World Models [93.07314830304193]
  Model-based reinforcement learning (MBRL) holds the promise of sample-efficient learning. We propose a simple yet effective approach, HarmonyDream, which automatically adjusts loss coefficients to maintain task harmonization.
  arXiv  Detail & Related papers  (2023-09-30T11:38:13Z)
• Causal Dynamics Learning for Task-Independent State Abstraction [61.707048209272884]
  We introduce Causal Dynamics Learning for Task-Independent State Abstraction (CDL) CDL learns a theoretically proved causal dynamics model that removes unnecessary dependencies between state variables and the action. A state abstraction can then be derived from the learned dynamics.
  arXiv  Detail & Related papers  (2022-06-27T17:02:53Z)
• DevFormer: A Symmetric Transformer for Context-Aware Device Placement [12.400790776196667]
  We present DevFormer, a transformer-based architecture for addressing the complex and computationally demanding problem of hardware design optimization. Our approach addresses this limitation by introducing strong inductive biases such as relative positional embeddings and action-permutation symmetricity. We show that DevFoemer outperforms state-of-the-art methods in both simulated hardware, leading to improved performances while reducing the number of components by more than $30$%.
  arXiv  Detail & Related papers  (2022-05-26T08:36:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.