Reinforcement Learning-Based Market Making as a Stochastic Control on Non-Stationary Limit Order Book Dynamics

• URL: http://arxiv.org/abs/2509.12456v1
• Date: Mon, 15 Sep 2025 21:08:13 GMT
• Title: Reinforcement Learning-Based Market Making as a Stochastic Control on Non-Stationary Limit Order Book Dynamics
• Authors: Rafael Zimmer, Oswaldo Luiz do Valle Costa,
• Abstract summary: Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies.<n>This paper explores the integration of a reinforcement learning agent in a market-making context.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies, enabling market makers to optimize decision-making policies based on interactions with the limit order book environment. This paper explores the integration of a reinforcement learning agent in a market-making context, where the underlying market dynamics have been explicitly modeled to capture observed stylized facts of real markets, including clustered order arrival times, non-stationary spreads and return drifts, stochastic order quantities and price volatility. These mechanisms aim to enhance stability of the resulting control agent, and serve to incorporate domain-specific knowledge into the agent policy learning process. Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent's performance under varying market conditions via a simulator-based environment. As evidenced by our analysis of the financial return and risk metrics when compared to a closed-form optimal solution, our results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Related papers

• Adaptive Financial Sentiment Analysis for NIFTY 50 via Instruction-Tuned LLMs , RAG and Reinforcement Learning Approaches [1.9116784879310027]
  Existing works in financial sentiment analysis have not considered the impact of stock prices or market feedback on sentiment analysis.<n>We propose an adaptive framework that integrates large language models (LLMs) with real-world stock market feedback to improve sentiment classification.
  arXiv  Detail & Related papers  (2025-12-23T06:27:12Z)
• Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets [57.179679246370114]
  In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices.<n>During deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact.<n>Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties.<n>We develop a novel class of elliptic uncertainty sets, enabling efficient and tractable robust policy evaluation.
  arXiv  Detail & Related papers  (2025-10-22T18:22:25Z)
• Trade in Minutes! Rationality-Driven Agentic System for Quantitative Financial Trading [57.28635022507172]
  TiMi is a rationality-driven multi-agent system that architecturally decouples strategy development from minute-level deployment.<n>We propose a two-tier analytical paradigm from macro patterns to micro customization, layered programming design for trading bot implementation, and closed-loop optimization driven by mathematical reflection.
  arXiv  Detail & Related papers  (2025-10-06T13:08:55Z)
• Market Making Strategies with Reinforcement Learning [0.0]
  Market makers (MMs) play a fundamental role in providing liquidity, yet face significant challenges arising from inventory risk, competition, and non-stationary market dynamics.<n>This research explores how Reinforcement Learning can be employed to develop autonomous, adaptive, and profitable market making strategies.
  arXiv  Detail & Related papers  (2025-07-24T16:17:49Z)
• Dynamic Reinsurance Treaty Bidding via Multi-Agent Reinforcement Learning [0.0]
  This paper develops a novel multi-agent reinforcement learning (MARL) framework for reinsurance treaty bidding.<n>MARL agents achieve up to 15% higher underwriting profit, 20% lower tail risk, and over 25% improvement in Sharpe ratios.<n>These findings suggest that MARL offers a viable path toward more transparent, adaptive, and risk-sensitive reinsurance markets.
  arXiv  Detail & Related papers  (2025-06-16T05:43:22Z)
• Cross-Asset Risk Management: Integrating LLMs for Real-Time Monitoring of Equity, Fixed Income, and Currency Markets [30.815524322885754]
  Large language models (LLMs) have emerged as powerful tools in the field of finance.<n>We introduce a Cross-Asset Risk Management framework that utilizes LLMs to facilitate real-time monitoring of equity, fixed income, and currency markets.
  arXiv  Detail & Related papers  (2025-04-05T22:28:35Z)
• When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments [55.19252983108372]
  We have developed a multi-agent AI system called StockAgent, driven by LLMs. The StockAgent allows users to evaluate the impact of different external factors on investor trading. It avoids the test set leakage issue present in existing trading simulation systems based on AI Agents.
  arXiv  Detail & Related papers  (2024-07-15T06:49:30Z)
• Insurance pricing on price comparison websites via reinforcement learning [7.023335262537794]
  This paper introduces reinforcement learning framework that learns optimal pricing policy by integrating model-based and model-free methods. The paper also highlights the importance of evaluating pricing policies using an offline dataset in a consistent fashion.
  arXiv  Detail & Related papers  (2023-08-14T04:44:56Z)
• Finding Regularized Competitive Equilibria of Heterogeneous Agent Macroeconomic Models with Reinforcement Learning [151.03738099494765]
  We study a heterogeneous agent macroeconomic model with an infinite number of households and firms competing in a labor market. We propose a data-driven reinforcement learning framework that finds the regularized competitive equilibrium of the model.
  arXiv  Detail & Related papers  (2023-02-24T17:16:27Z)
• A simple learning agent interacting with an agent-based market model [0.0]
  We consider the learning dynamics of a single reinforcement learning optimal execution trading agent when it interacts with an agent-based financial market model. We find that the moments of the model are robust to the impact of the learning agents except for the Hurst exponent. The introduction of the learning agent preserves the shape of the price impact curves but can reduce the trade-sign auto-correlations when their trading volumes increase.
  arXiv  Detail & Related papers  (2022-08-22T16:42:06Z)
• A Modular Framework for Reinforcement Learning Optimal Execution [68.8204255655161]
  We develop a modular framework for the application of Reinforcement Learning to the problem of Optimal Trade Execution. The framework is designed with flexibility in mind, in order to ease the implementation of different simulation setups.
  arXiv  Detail & Related papers  (2022-08-11T09:40:42Z)
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation [93.52573037053449]
  H-MARL (Hallucinated Multi-Agent Reinforcement Learning) learns successful equilibrium policies after a few interactions with the environment. We demonstrate our approach experimentally on an autonomous driving simulation benchmark.
  arXiv  Detail & Related papers  (2022-03-14T17:24:03Z)
• Universal Trading for Order Execution with Oracle Policy Distillation [99.57416828489568]
  We propose a novel universal trading policy optimization framework to bridge the gap between the noisy yet imperfect market states and the optimal action sequences for order execution. We show that our framework can better guide the learning of the common policy towards practically optimal execution by an oracle teacher with perfect information.
  arXiv  Detail & Related papers  (2021-01-28T05:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.