Robust Market Making via Adversarial Reinforcement Learning
- URL: http://arxiv.org/abs/2003.01820v2
- Date: Wed, 8 Jul 2020 15:15:09 GMT
- Title: Robust Market Making via Adversarial Reinforcement Learning
- Authors: Thomas Spooner, Rahul Savani
- Abstract summary: We show that adversarial reinforcement learning can be used to produce market marking agents robust to adversarial and adaptively-chosen market conditions.
We show that our ARL method consistently converges, and we prove for several special cases that the profiles that we converge to correspond to Nash equilibria in a simplified single-stage game.
- Score: 5.482532589225552
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We show that adversarial reinforcement learning (ARL) can be used to produce
market marking agents that are robust to adversarial and adaptively-chosen
market conditions. To apply ARL, we turn the well-studied single-agent model of
Avellaneda and Stoikov [2008] into a discrete-time zero-sum game between a
market maker and adversary. The adversary acts as a proxy for other market
participants that would like to profit at the market maker's expense. We
empirically compare two conventional single-agent RL agents with ARL, and show
that our ARL approach leads to: 1) the emergence of risk-averse behaviour
without constraints or domain-specific penalties; 2) significant improvements
in performance across a set of standard metrics, evaluated with or without an
adversary in the test environment, and; 3) improved robustness to model
uncertainty. We empirically demonstrate that our ARL method consistently
converges, and we prove for several special cases that the profiles that we
converge to correspond to Nash equilibria in a simplified single-stage game.
Related papers
- Deviations from the Nash equilibrium and emergence of tacit collusion in a two-player optimal execution game with reinforcement learning [0.9208007322096533]
We study a scenario in which two autonomous agents learn to liquidate the same asset optimally in the presence of market impact.
Our results show that the strategies learned by the agents deviate significantly from the Nash equilibrium of the corresponding market impact game.
We explore how different levels of market volatility influence the agents' performance and the equilibria they discover.
arXiv Detail & Related papers (2024-08-21T16:54:53Z) - Toward Optimal LLM Alignments Using Two-Player Games [86.39338084862324]
In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent.
We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents.
Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.
arXiv Detail & Related papers (2024-06-16T15:24:50Z) - Provable Unrestricted Adversarial Training without Compromise with Generalizability [44.02361569894942]
Adversarial training (AT) is widely considered as the most promising strategy to defend against adversarial attacks.
The existing AT methods often achieve adversarial robustness at the expense of standard generalizability.
We propose a novel AT approach called Provable Unrestricted Adversarial Training (PUAT)
arXiv Detail & Related papers (2023-01-22T07:45:51Z) - Towards Multi-Agent Reinforcement Learning driven Over-The-Counter
Market Simulations [16.48389671789281]
We study a game between liquidity provider and liquidity taker agents interacting in an over-the-counter market.
By playing against each other, our deep-reinforcement-learning-driven agents learn emergent behaviors.
We show convergence rates for our multi-agent policy gradient algorithm under a transitivity assumption.
arXiv Detail & Related papers (2022-10-13T17:06:08Z) - Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment.
We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent.
We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
arXiv Detail & Related papers (2022-09-24T14:22:29Z) - Deep Q-Learning Market Makers in a Multi-Agent Simulated Stock Market [58.720142291102135]
This paper focuses precisely on the study of these markets makers strategies from an agent-based perspective.
We propose the application of Reinforcement Learning (RL) for the creation of intelligent market markers in simulated stock markets.
arXiv Detail & Related papers (2021-12-08T14:55:21Z) - Learning who is in the market from time series: market participant
discovery through adversarial calibration of multi-agent simulators [0.0]
In electronic trading markets only the price or volume time series are directly observable.
We propose a novel two-step method to train a discriminator that is able to distinguish between "real" and "fake" price and volume time series.
arXiv Detail & Related papers (2021-08-02T06:53:37Z) - Explore and Control with Adversarial Surprise [78.41972292110967]
Reinforcement learning (RL) provides a framework for learning goal-directed policies given user-specified rewards.
We propose a new unsupervised RL technique based on an adversarial game which pits two policies against each other to compete over the amount of surprise an RL agent experiences.
We show that our method leads to the emergence of complex skills by exhibiting clear phase transitions.
arXiv Detail & Related papers (2021-07-12T17:58:40Z) - Robust Reinforcement Learning on State Observations with Learned Optimal
Adversary [86.0846119254031]
We study the robustness of reinforcement learning with adversarially perturbed state observations.
With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found.
For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones.
arXiv Detail & Related papers (2021-01-21T05:38:52Z) - Robust Reinforcement Learning using Adversarial Populations [118.73193330231163]
Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness.
We show that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary.
We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training.
arXiv Detail & Related papers (2020-08-04T20:57:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.