Robust Control under Stationary Ambiguity

• URL: http://arxiv.org/abs/2608.04832v1
• Date: Wed, 05 Aug 2026 13:34:00 GMT
• Title: Robust Control under Stationary Ambiguity
• Abstract summary: Control policies optimized in simulation can perform poorly in the real system when the parameters of the simulator are estimated from limited data.<n>We propose training policies in simulators where ambiguity varies with the system's state but does not systematically decay over time.<n>We show how to construct such simulators and demonstrate, on hedging problems, that policies trained under stationary ambiguity preserve robustness to latent factors over time.
• Score: 5.394466899760558
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Control policies optimized in simulation can perform poorly in the real system when the parameters $x$ of the simulator are estimated from limited data but the resulting parameter uncertainty is not represented inside the simulation. A common way to incorporate such ambiguity is to simulate each trajectory of the system under a randomly drawn value for $x$. Since the policy cannot observe the drawn value, it must initially choose controls that perform well across many possible parameter values. However, if the policy progressively observes the system, it can often gradually infer the value of $x$, so that ambiguity vanishes. Over time, the policy then specializes to its estimate of $x$ and loses its robustness. This is undesirable in many real systems, where latent factors are expected to shift. In financial markets, for example, a policy hedging a derivative payoff should remain robust to changes in the volatility regime. To induce such continual robustness, we propose training policies in simulators where ambiguity varies with the system's state but does not systematically decay over time. We formalize this requirement as stationary ambiguity: the simulator should induce a stationary filter process over the latent state. We show how to construct such simulators and demonstrate, on hedging problems, that policies trained under stationary ambiguity preserve robustness to latent factors over time, leading to strong performance on real market data. As a modeling principle, stationary ambiguity informs many simulator design decisions: which models make realistic simulators, how their parameters should be randomized, and how simulator and policy should be initialized. While our experiments focus on hedging, stationary ambiguity may also be useful for other sequential control problems driven by exogenous stochastic processes with shifting latent structure.

Related papers

• How Should a Simulation-to-Reality Transfer Budget Be Spent? [0.0]
  We focus on the allocation of real-robot measurement time between system identification and domain randomization.<n>Across the reality gaps and noise levels we tested, the measurement budget did most of the work.<n>Overall, our results suggest that sim-to-real pipelines should first measure the parameters they can and reserve randomization for the uncertainty that remains.
  arXiv  Detail & Related papers  (2026-06-20T14:23:34Z)
• Sim2Act: Robust Simulation-to-Decision Learning via Adversarial Calibration and Group-Relative Perturbation [54.29523408543184]
  Simulation-to-decision learning enables safe policy training in digital environments without risking real-world deployment.<n>Existing approaches either focus on improving average simulation fidelity or adopt conservative regularization.<n>We propose Sim2Act, a robust simulation-to-decision framework that addresses both simulator and policy robustness.
  arXiv  Detail & Related papers  (2026-03-10T00:51:47Z)
• ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation [72.78362530982109]
  ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation, is a framework that decouples exploration from commitment.<n>We show that naive LLM-based simulators struggle to capture rare but high-impact failure modes.<n>We introduce a risk-aware tool simulator that emphasizes fidelity on failure-inducing actions.
  arXiv  Detail & Related papers  (2026-02-02T06:33:22Z)
• Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators [9.868826622074899]
  SureSim is a framework to augment large-scale simulation with relatively small-scale real-world testing.<n>We leverage non-asymptotic mean estimation algorithms to provide confidence intervals on mean policy performance.<n>Our approach saves over (20-25%) of hardware evaluation effort to achieve similar bounds on policy performance.
  arXiv  Detail & Related papers  (2025-10-05T20:37:53Z)
• G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration [48.948187359727996]
  G-Sim is a hybrid framework that automates simulator construction with rigorous empirical calibration.<n>It produces reliable, causally-informed simulators, mitigating data-inefficiency and enabling robust system-level interventions.
  arXiv  Detail & Related papers  (2025-06-10T22:14:34Z)
• Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics [3.7491742648742568]
  Domain randomization is a technique to facilitate the transfer of policies from simulation to real-world robotic applications.<n>We propose a method to enable safe deployment-time policy adaptation in real-world robot control.
  arXiv  Detail & Related papers  (2025-03-13T23:28:11Z)
• Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
  Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
  arXiv  Detail & Related papers  (2023-07-28T05:47:24Z)
• Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model [42.28001762749647]
  In high-stake scenarios like medical treatment and auto-piloting, it's risky or even infeasible to collect online experimental data to train the agent. We consider policy learning for Robust Markov Decision Processes (RMDP), where the agent tries to seek a robust policy with respect to unexpected perturbations on the environments. Our goal is to identify a near-optimal robust policy for the perturbed testing environment, which introduces additional technical difficulties.
  arXiv  Detail & Related papers  (2022-03-13T06:37:25Z)
• Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization [63.75188254377202]
  Deep reinforcement learning algorithms can perform poorly in real-world tasks due to discrepancy between source and target environments. We propose a novel model-free actor-critic algorithm to learn robust policies without modeling the disturbance in advance. Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.
  arXiv  Detail & Related papers  (2021-12-20T13:13:05Z)
• Robust Value Iteration for Continuous Control Tasks [99.00362538261972]
  When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well. We present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain. We show that robust value is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm.
  arXiv  Detail & Related papers  (2021-05-25T19:48:35Z)
• Auto-Tuned Sim-to-Real Transfer [143.44593793640814]
  Policies trained in simulation often fail when transferred to the real world. Current approaches to tackle this problem, such as domain randomization, require prior knowledge and engineering. We propose a method for automatically tuning simulator system parameters to match the real world.
  arXiv  Detail & Related papers  (2021-04-15T17:59:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.