Markov Decision Processes under Model Uncertainty
- URL: http://arxiv.org/abs/2206.06109v1
- Date: Mon, 13 Jun 2022 12:51:31 GMT
- Title: Markov Decision Processes under Model Uncertainty
- Authors: Ariel Neufeld, Julian Sester, Mario \v{S}iki\'c
- Abstract summary: We introduce a general framework for Markov decision problems under model uncertainty.
We apply this framework to portfolio optimization involving data of the S&P 500.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce a general framework for Markov decision problems under model
uncertainty in a discrete-time infinite horizon setting. By providing a dynamic
programming principle we obtain a local-to-global paradigm, namely solving a
local, i.e., a one time-step robust optimization problem leads to an optimizer
of the global (i.e. infinite time-steps) robust stochastic optimal control
problem, as well as to a corresponding worst-case measure. Moreover, we apply
this framework to portfolio optimization involving data of the S&P 500. We
present two different types of ambiguity sets; one is fully data-driven given
by a Wasserstein-ball around the empirical measure, the second one is described
by a parametric set of multivariate normal distributions, where the
corresponding uncertainty sets of the parameters are estimated from the data.
It turns out that in scenarios where the market is volatile or bearish, the
optimal portfolio strategies from the corresponding robust optimization problem
outperforms the ones without model uncertainty, showcasing the importance of
taking model uncertainty into account.
Related papers
- End-to-End Learning for Fair Multiobjective Optimization Under
Uncertainty [55.04219793298687]
The Predict-Then-Forecast (PtO) paradigm in machine learning aims to maximize downstream decision quality.
This paper extends the PtO methodology to optimization problems with nondifferentiable Ordered Weighted Averaging (OWA) objectives.
It shows how optimization of OWA functions can be effectively integrated with parametric prediction for fair and robust optimization under uncertainty.
arXiv Detail & Related papers (2024-02-12T16:33:35Z) - Model-Free Robust Average-Reward Reinforcement Learning [25.125481838479256]
We focus on the robust average-reward MDPs under the model-free iteration setting.
We design two model-free algorithms, robust relative value (RVI) TD and robust RVI Q-learning, and theoretically prove their convergence to the optimal solution.
arXiv Detail & Related papers (2023-05-17T18:19:23Z) - Improving Generalization via Uncertainty Driven Perturbations [107.45752065285821]
We consider uncertainty-driven perturbations of the training data points.
Unlike loss-driven perturbations, uncertainty-guided perturbations do not cross the decision boundary.
We show that UDP is guaranteed to achieve the robustness margin decision on linear models.
arXiv Detail & Related papers (2022-02-11T16:22:08Z) - Sample Complexity of Robust Reinforcement Learning with a Generative
Model [0.0]
We propose a model-based reinforcement learning (RL) algorithm for learning an $epsilon$-optimal robust policy.
We consider three different forms of uncertainty sets, characterized by the total variation distance, chi-square divergence, and KL divergence.
In addition to the sample complexity results, we also present a formal analytical argument on the benefit of using robust policies.
arXiv Detail & Related papers (2021-12-02T18:55:51Z) - Integrated Conditional Estimation-Optimization [6.037383467521294]
Many real-world optimization problems uncertain parameters with probability can be estimated using contextual feature information.
In contrast to the standard approach of estimating the distribution of uncertain parameters, we propose an integrated conditional estimation approach.
We show that our ICEO approach is theally consistent under moderate conditions.
arXiv Detail & Related papers (2021-10-24T04:49:35Z) - Robustifying Conditional Portfolio Decisions via Optimal Transport [19.30689697702845]
We propose a data-driven portfolio selection model that integrates side information, conditional estimation and robustness.
We show that the distributionally robust portfolio allocation with side information problem can be reformulated as a finite-dimensional optimization problem.
arXiv Detail & Related papers (2021-03-30T15:56:03Z) - Modeling the Second Player in Distributionally Robust Optimization [90.25995710696425]
We argue for the use of neural generative models to characterize the worst-case distribution.
This approach poses a number of implementation and optimization challenges.
We find that the proposed approach yields models that are more robust than comparable baselines.
arXiv Detail & Related papers (2021-03-18T14:26:26Z) - Stein Variational Model Predictive Control [130.60527864489168]
Decision making under uncertainty is critical to real-world, autonomous systems.
Model Predictive Control (MPC) methods have demonstrated favorable performance in practice, but remain limited when dealing with complex distributions.
We show that this framework leads to successful planning in challenging, non optimal control problems.
arXiv Detail & Related papers (2020-11-15T22:36:59Z) - Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
We show that common optimization methods lead to poor variational approximations if the problem is moderately large.
Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
arXiv Detail & Related papers (2020-09-01T19:12:11Z) - Accounting for Unobserved Confounding in Domain Generalization [107.0464488046289]
This paper investigates the problem of learning robust, generalizable prediction models from a combination of datasets.
Part of the challenge of learning robust models lies in the influence of unobserved confounders.
We demonstrate the empirical performance of our approach on healthcare data from different modalities.
arXiv Detail & Related papers (2020-07-21T08:18:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.