Plan Before You Trade: Inference-Time Optimization for RL Trading Agents

• URL: http://arxiv.org/abs/2605.12653v1
• Date: Tue, 12 May 2026 18:58:03 GMT
• Title: Plan Before You Trade: Inference-Time Optimization for RL Trading Agents
• Abstract summary: Reinforcement learning agents for portfolio management are typically trained and deployed as static policies.<n>We propose a plugin inference-time optimization framework inspired by Model Predictive Control (MPC)<n>Our framework is compatible with any pre-trained agent and adapts the policy to the forecaster's predictions without any retraining.
• Score: 8.39123686251568
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning agents for portfolio management are typically trained and deployed as static policies, with no mechanism for using price forecasts at inference time. We propose $\text{FPILOT}$ (**Fin**ancial **P**lugin **I**nference-time **L**earning for **O**ptimal **T**rading), a plugin inference-time optimization framework inspired by Model Predictive Control (MPC). Our key structural insight is that future prices mostly do not depend on one agent's portfolio allocation, so a suitable predictive model can produce a multi-step price trajectory without iterative action-conditioned rollouts as in typical reinforcement learning. At each decision step, we use the forecaster's predicted price trajectory to construct an allocation-based imagined return objective, and optimize the policy at inference-time before executing one step of the trade. Our framework is compatible with any pre-trained agent and adapts the policy to the forecaster's predictions without any retraining. Evaluated across five policy learning algorithms on the TradeMaster DJ30 benchmark, $\text{FPILOT}$ produces consistent improvements in total return and return-based risk-adjusted metrics (Sharpe, Sortino, Calmar), with stochastic policies benefiting more than deterministic ones. Further, using synthetic forecasts at calibrated quality levels, we show that gains consistently improve with forecaster quality, suggesting that our performance will improve based on advances in financial forecasting.

Related papers

• Decision-focused Sparse Tangent Portfolio Optimization [32.04232046262782]
  convex programming layer replaces discrete selection with a smooth top-$k$ operator enforcing an exact cardinality $k$.<n>This enables flow through prediction, asset selection, and re-optimization, allowing the predictive model to directly optimize portfolio performance.<n>Across four major equity markets, our method achieves competitive and often superior out-of-sample Sharpe ratios.
  arXiv  Detail & Related papers  (2026-07-01T08:06:07Z)
• Deployment-Side Adaptiveness in Multi-Horizon Volatility Forecasting [1.2489632787815885]
  We study how a trained multi-output (MIMO) forecaster induces a family of forecasts with different accuracy and cost profiles.<n>Across 20 stock-volatility series, three forecast horizons, and architectures ranging from linear models to PatchTST, we find that non-default rollout rules often improve.<n>We evaluate validation-based deployment policies over the induced rule family.
  arXiv  Detail & Related papers  (2026-06-26T03:32:09Z)
• Ratio-Variance Regularized Policy Optimization [64.95520246570446]
  We show that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints.<n>We introduce $bf R2bf VPO$ (Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework.
  arXiv  Detail & Related papers  (2026-05-26T09:53:42Z)
• $V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts [81.48669089692189]
  Generalist Value Models (such as $V_0.5$) achieve pre-trained value estimation by explicitly encoding model capabilities in-context.<n>We propose $V_0.5$, which adaptively fuses the baseline predicted by such value model with the empirical mean derived from sparse rollouts.<n>$V_0.5$ significantly outperforms GRPO and DAPO, achieving faster convergence and over some 10% performance improvement.
  arXiv  Detail & Related papers  (2026-03-11T14:57:41Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• ProAct: Agentic Lookahead in Interactive Environments [56.50613398808361]
  ProAct is a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm.<n>We introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search.<n>We also propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms.
  arXiv  Detail & Related papers  (2026-02-05T05:45:16Z)
• $V_0$: A Generalist Value Model for Any Policy at State Zero [80.7505802128501]
  Policy methods rely on a baseline to measure the relative advantage of an action.<n>This baseline is typically estimated by a Value Model (Critic) often as large as the policy model itself.<n>We propose a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts.
  arXiv  Detail & Related papers  (2026-02-03T14:35:23Z)
• Generative Actor Critic [74.04971271003869]
  Generative Actor Critic (GAC) is a novel framework that decouples sequential decision-making by reframing textitpolicy evaluation as learning a generative model of the joint distribution over trajectories and returns.<n>Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-12-25T06:31:11Z)
• Integrated Prediction and Multi-period Portfolio Optimization [29.582959310549594]
  Multi-period portfolio optimization accounts for transaction costs, path-dependent risks, and the intertemporal structure of trading decisions.<n>This paper introduces IPMO, a model for multi-period mean-variance portfolio optimization with turnover penalties.<n>For scalability, we introduce a mirror-descent fixed-point (MDFP) differentiation scheme that avoids factorizing the Karush-Kuhn-Tucker (KKT) systems.
  arXiv  Detail & Related papers  (2025-12-12T04:31:22Z)
• Intelligent Routing for Sparse Demand Forecasting: A Comparative Evaluation of Selection Strategies [0.6798775532273751]
  parse and intermittent demand forecasting in supply chains presents a critical challenge.<n>We propose a Model-spanning framework that selects the most suitable forecasting model-spanning classical, ML, and DL methods for each product.<n>Experiments on the large-scale Favorita dataset show our deep learning (Inception Time) router improves forecasting accuracy by up to 11.8%.
  arXiv  Detail & Related papers  (2025-06-04T03:09:45Z)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression [52.0792918455501]
  We propose a novel two-stage policy optimization framework that directly approximates the optimal advantage function.<n>$A$*-PO achieves competitive performance across a wide range of mathematical reasoning benchmarks.<n>It reduces training time by up to 2$times$ and peak memory usage by over 30% compared to PPO, GRPO, and REBEL.
  arXiv  Detail & Related papers  (2025-05-27T03:58:50Z)
• Structured Dynamic Pricing: Optimal Regret in a Global Shrinkage Model [50.06663781566795]
  We consider a dynamic model with the consumers' preferences as well as price sensitivity varying over time. We measure the performance of a dynamic pricing policy via regret, which is the expected revenue loss compared to a clairvoyant that knows the sequence of model parameters in advance. Our regret analysis results not only demonstrate optimality of the proposed policy but also show that for policy planning it is essential to incorporate available structural information.
  arXiv  Detail & Related papers  (2023-03-28T00:23:23Z)
• Deep Reinforcement Learning for Stock Portfolio Optimization [0.0]
  We will formulate the problem such that we can apply Reinforcement Learning for the task properly. To maintain a realistic assumption about the market, we will incorporate transaction cost and risk factor into the state as well. We will present the end-to-end solution for the task with Minimum Variance Portfolio for stock subset selection, and Wavelet Transform for extracting multi-frequency data pattern.
  arXiv  Detail & Related papers  (2020-12-09T10:19:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.