Can Reinforcement Learning Efficiently Discover Price Manipulation?

• URL: http://arxiv.org/abs/2607.06121v1
• Date: Tue, 07 Jul 2026 10:29:20 GMT
• Title: Can Reinforcement Learning Efficiently Discover Price Manipulation?
• Abstract summary: We consider a single-asset market in which prices evolve according to an Almgren-Chriss framework with non-linear permanent impact and linear temporary impact.<n>We first establish the existence of price-manipulative strategies in discrete time and compute the optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information.<n>We then compare two finite-sample learning approaches: a model-based procedure that estimates impact parameters from simulated execution data and an RL approach based on Deep Deterministic Policy Gradient, trained directly on the same amount of data.
• Score: 0.254890465057467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we investigate whether a model-free RL agent can identify and exploit price manipulation opportunities more effectively than a traditional model-based approach that assumes correct specification of the data-generating process but relies on noisy parameter estimates. We consider a single-asset market in which prices evolve according to an Almgren-Chriss framework with non-linear permanent impact and linear temporary impact. We first establish the existence of price-manipulative strategies in discrete time and compute the optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information. We then compare two finite-sample learning approaches: a model-based procedure that estimates impact parameters from simulated execution data and an agnostic RL approach based on Deep Deterministic Policy Gradient, trained directly on the same amount of data. For intermediate volatility, the RL agent successfully discovers profitable manipulative strategies without explicit knowledge of the underlying model, even when training data are quite limited. More importantly, RL consistently outperforms the model-based approach when parameter estimates are affected by sampling error, despite the latter benefiting from the correct model specification. For large volatility, all methods are unable to identify manipulation opportunities, while for small volatility, the model based approach outperforms RL. These findings highlight both the effectiveness of RL in complex control problems and the risks associated with deploying learning algorithms in financial markets without appropriate safeguards.

Related papers

• Uncertainty-Aware Strategies: A Model-Agnostic Framework for Robust Financial Optimization through Subsampling [0.7916373508978822]
  This paper addresses the challenge of model uncertainty in quantitative finance.<n>Decisions in portfolio allocation, derivative pricing, and risk management rely on estimating models from limited data.<n>We superimpose an outer "uncertainty measure", motivated by traditional monetary risk measures, on the space of models.
  arXiv  Detail & Related papers  (2025-06-08T21:55:00Z)
• VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning [15.796963385626846]
  VIPO is a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training.<n>We show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods.
  arXiv  Detail & Related papers  (2025-04-16T10:23:44Z)
• Look Before Leap: Look-Ahead Planning with Uncertainty in Reinforcement Learning [4.902161835372679]
  We propose a novel framework for uncertainty-aware policy optimization with model-based exploratory planning.<n>In the policy optimization phase, we leverage an uncertainty-driven exploratory policy to actively collect diverse training samples.<n>Our approach offers flexibility and applicability to tasks with varying state/action spaces and reward structures.
  arXiv  Detail & Related papers  (2025-03-26T01:07:35Z)
• Commodities Trading through Deep Policy Gradient Methods [0.0]
  It formulates the commodities trading problem as a continuous, discrete-time dynamical system. Two policy algorithms, namely actor-based and actor-critic-based approaches, are introduced. Backtesting on front-month natural gas futures demonstrates that DRL models increase the Sharpe ratio by $83%$ compared to the buy-and-hold baseline.
  arXiv  Detail & Related papers  (2023-08-10T17:21:12Z)
• Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization [41.774837419584735]
  offline reinforcement learning (RL) addresses the problem of learning a performant policy from a fixed batch of data collected by following some behavior policy. Model-based approaches are particularly appealing since they can extract more learning signals from the logged dataset by learning a model of the environment.
  arXiv  Detail & Related papers  (2022-10-07T20:13:50Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• On Effective Scheduling of Model-based Reinforcement Learning [53.027698625496015]
  We propose a framework named AutoMBPO to automatically schedule the real data ratio. In this paper, we first theoretically analyze the role of real data in policy training, which suggests that gradually increasing the ratio of real data yields better performance.
  arXiv  Detail & Related papers  (2021-11-16T15:24:59Z)
• DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation [57.358212277226315]
  In imitation learning from observation IfO, a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator. Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms. This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk. We propose a more data-efficient IfO algorithm
  arXiv  Detail & Related papers  (2021-03-31T23:46:32Z)
• COMBO: Conservative Offline Model-Based Policy Optimization [120.55713363569845]
  Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We develop a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-actions. We find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods.
  arXiv  Detail & Related papers  (2021-02-16T18:50:32Z)
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning [93.1435980666675]
  We show how optimistic exploration can be easily combined with state-of-the-art reinforcement learning algorithms. Our experiments demonstrate that optimistic exploration significantly speeds-up learning when there are penalties on actions.
  arXiv  Detail & Related papers  (2020-06-15T18:37:38Z)
• MOPO: Model-based Offline Policy Optimization [183.6449600580806]
  offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. We show that an existing model-based RL algorithm already produces significant gains in the offline setting. We propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics.
  arXiv  Detail & Related papers  (2020-05-27T08:46:41Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.