IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making

• URL: http://arxiv.org/abs/2308.08918v1
• Date: Thu, 17 Aug 2023 11:04:09 GMT
• Title: IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making
• Authors: Hui Niu, Siyuan Li, Jiahao Zheng, Zhouchi Lin, Jian Li, Jian Guo, Bo An
• Abstract summary: Reinforcement Learning technology has achieved remarkable success in quantitative trading. Most existing RL-based market making methods focus on optimizing single-price level strategies. We propose Imitative Market Maker (IMM), a novel RL framework leveraging both knowledge from suboptimal signal-based experts and direct policy interactions.
• Score: 33.23156884634365
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Market making (MM) has attracted significant attention in financial trading owing to its essential function in ensuring market liquidity. With strong capabilities in sequential decision-making, Reinforcement Learning (RL) technology has achieved remarkable success in quantitative trading. Nonetheless, most existing RL-based MM methods focus on optimizing single-price level strategies which fail at frequent order cancellations and loss of queue priority. Strategies involving multiple price levels align better with actual trading scenarios. However, given the complexity that multi-price level strategies involves a comprehensive trading action space, the challenge of effectively training profitable RL agents for MM persists. Inspired by the efficient workflow of professional human market makers, we propose Imitative Market Maker (IMM), a novel RL framework leveraging both knowledge from suboptimal signal-based experts and direct policy interactions to develop multi-price level MM strategies efficiently. The framework start with introducing effective state and action representations adept at encoding information about multi-price level orders. Furthermore, IMM integrates a representation learning unit capable of capturing both short- and long-term market trends to mitigate adverse selection risk. Subsequently, IMM formulates an expert strategy based on signals and trains the agent through the integration of RL and imitation learning techniques, leading to efficient learning. Extensive experimental results on four real-world market datasets demonstrate that IMM outperforms current RL-based market making strategies in terms of several financial criteria. The findings of the ablation study substantiate the effectiveness of the model components.

Related papers

• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Automate Strategy Finding with LLM in Quant investment [4.46212317245124]
  We propose a novel framework for quantitative stock investment in portfolio management and alpha mining. This paper proposes a framework where large language models (LLMs) mine alpha factors from multimodal financial data. Experiments on the Chinese stock markets demonstrate that this framework significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-09-10T07:42:28Z)
• When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments [55.19252983108372]
  We have developed a multi-agent AI system called StockAgent, driven by LLMs. The StockAgent allows users to evaluate the impact of different external factors on investor trading. It avoids the test set leakage issue present in existing trading simulation systems based on AI Agents.
  arXiv  Detail & Related papers  (2024-07-15T06:49:30Z)
• Learning the Market: Sentiment-Based Ensemble Trading Agents [5.005352154557397]
  We propose and study the integration of sentiment analysis and deep reinforcement learning ensemble algorithms for stock trading. We show that our approach results in a strategy that is profitable, robust, and risk-minimal.
  arXiv  Detail & Related papers  (2024-02-02T14:34:22Z)
• Developing A Multi-Agent and Self-Adaptive Framework with Deep Reinforcement Learning for Dynamic Portfolio Risk Management [1.2016264781280588]
  A multi-agent reinforcement learning (RL) approach is proposed to balance the trade-off between the overall portfolio returns and their potential risks. The obtained empirical results clearly reveal the potential strengths of our proposed MASA framework.
  arXiv  Detail & Related papers  (2024-02-01T11:31:26Z)
• Harnessing Deep Q-Learning for Enhanced Statistical Arbitrage in High-Frequency Trading: A Comprehensive Exploration [0.0]
  Reinforcement Learning (RL) is a branch of machine learning where agents learn by interacting with their environment. This paper dives deep into the integration of RL in statistical arbitrage strategies tailored for High-Frequency Trading (HFT) scenarios. Through extensive simulations and backtests, our research reveals that RL not only enhances the adaptability of trading strategies but also shows promise in improving profitability metrics and risk-adjusted returns.
  arXiv  Detail & Related papers  (2023-09-13T06:15:40Z)
• Factor Investing with a Deep Multi-Factor Model [123.52358449455231]
  We develop a novel deep multi-factor model that adopts industry neutralization and market neutralization modules with clear financial insights. Tests on real-world stock market data demonstrate the effectiveness of our deep multi-factor model.
  arXiv  Detail & Related papers  (2022-10-22T14:47:11Z)
• MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization [17.759687104376855]
  We propose a novel two-stage-based approach for portfolio management. In the first stage, incorporates an imitation learning into the reinforcement learning framework. In the second stage, learns a meta-policy to recognize the market conditions and decide on the most proper learned policy to follow.
  arXiv  Detail & Related papers  (2022-09-01T07:58:06Z)
• Deep Q-Learning Market Makers in a Multi-Agent Simulated Stock Market [58.720142291102135]
  This paper focuses precisely on the study of these markets makers strategies from an agent-based perspective. We propose the application of Reinforcement Learning (RL) for the creation of intelligent market markers in simulated stock markets.
  arXiv  Detail & Related papers  (2021-12-08T14:55:21Z)
• Universal Trading for Order Execution with Oracle Policy Distillation [99.57416828489568]
  We propose a novel universal trading policy optimization framework to bridge the gap between the noisy yet imperfect market states and the optimal action sequences for order execution. We show that our framework can better guide the learning of the common policy towards practically optimal execution by an oracle teacher with perfect information.
  arXiv  Detail & Related papers  (2021-01-28T05:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.