Decomposable Reward Modeling and Realistic Environment Design for Reinforcement Learning-Based Forex Trading

• URL: http://arxiv.org/abs/2604.00031v1
• Date: Fri, 20 Mar 2026 13:38:19 GMT
• Title: Decomposable Reward Modeling and Realistic Environment Design for Reinforcement Learning-Based Forex Trading
• Abstract summary: This paper presents a modular reinforcement learning framework for Forex trading.<n>It includes a friction-aware execution engine that enforces strict anti-lookahead semantics, with observations at time t, execution at time t+1, and mark-to-market at time t+1.<n>It also incorporates realistic costs such as spread, commission, slippage, rollover financing, and margin-triggered liquidation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Applying reinforcement learning (RL) to foreign exchange (Forex) trading remains challenging because realistic environments, well-defined reward functions, and expressive action spaces must be satisfied simultaneously, yet many prior studies rely on simplified simulators, single scalar rewards, and restricted action representations, limiting both interpretability and practical relevance. This paper presents a modular RL framework designed to address these limitations through three tightly integrated components: a friction-aware execution engine that enforces strict anti-lookahead semantics, with observations at time t, execution at time t+1, and mark-to-market at time t+1, while incorporating realistic costs such as spread, commission, slippage, rollover financing, and margin-triggered liquidation; a decomposable 11-component reward architecture with fixed weights and per-step diagnostic logging to enable systematic ablation and component-level attribution; and a 10-action discrete interface with legal-action masking that encodes explicit trading primitives while enforcing margin-aware feasibility constraints. Empirical evaluation on EURUSD focuses on learning dynamics rather than generalization and reveals strongly non-monotonic reward interactions, where additional penalties do not reliably improve outcomes; the full reward configuration achieves the highest training Sharpe (0.765) and cumulative return (57.09 percent). The expanded action space increases return but also turnover and reduces Sharpe relative to a conservative 3-action baseline, indicating a return-activity trade-off under a fixed training budget, while scaling-enabled variants consistently reduce drawdown, with the combined configuration achieving the strongest endpoint performance.

Related papers

• ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning [53.17987412112712]
  Reinforcement Learning (RL) has demonstrated significant potential for improving Vision-Language-Action (VLA) models on complex manipulation tasks.<n>In this work, we first investigate the exploration stagnation bottleneck in current VLA-RL frameworks.<n>We introduce RL Exploration Token (ExToken), a simple yet general framework that condition VLA policies on discrete behavioral priors.
  arXiv  Detail & Related papers  (2026-07-14T16:04:41Z)
• A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctions [19.323612677924732]
  We introduce the A3M framework, which integrates adaptive deep reinforcement learning, explicit adversarial reasoning, and principled multi-objective reward design for online auction strategy optimization.<n>A3M employs an actor-critic DRL backbone to dynamically balance exploration and exploitation, an opponent model for fictitious play against non-stationary adversaries, and a composite reward function to jointly maximize utility, auctioneer revenue, and fairness.<n>Results show that A3M reduces final regret by 30--40% in standard settings, maintains robust performance against adversarial strategy shifts, scales favorably with the number of units $K$, and enables tunable multi-objective trade
  arXiv  Detail & Related papers  (2026-06-27T14:30:43Z)
• PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models [50.232333672172395]
  Vision-Language-Action (VLA) models provide a unified paradigm for robotic manipulation, yet their real-world deployment is often bottlenecked by execution efficiency.<n>We propose textbfPolicyTrim, a reinforcement learning-based post-training framework that extends the reliable action chunk length and reduces redundant physical steps.<n>Our framework delivers up to a 5.83$times$ end-to-end deployment speedup without compromising task success rates.
  arXiv  Detail & Related papers  (2026-06-21T14:54:07Z)
• YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition [78.52944812151664]
  Large language models (LLMs) drive significant financial innovations, yet their deployment is bottlenecked by KV cache memory overhead.<n>We propose YouZhi-LLM, a highly efficient financial LLM empowered by a comprehensive structural transition and training pipeline built on the Huawei Ascend ecosystem.
  arXiv  Detail & Related papers  (2026-06-04T08:44:37Z)
• Training Language Models for Bilateral Trade with Private Information [47.25254982913256]
  Bilateral trade demands individual, strategic surplus rationality, and cooperation to realize gains from trade.<n>We develop a structured bargaining environment where large language models negotiate via tool calls within an event-driven simulator.<n>The environment serves two purposes: a benchmark for frontier models and as a training environment for open-weight models via reinforcement learning.
  arXiv  Detail & Related papers  (2026-04-10T03:04:20Z)
• Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning [74.5532558466687]
  Group Relative Reward Rescaling (GR$3$) is a general, continuous, and reward-dependent gating mechanism.<n> GR$3$maintains training dynamics and downstream performance comparable to standard GRPO.<n>It significantly mitigates length inflation, outperforming state-of-the-art length-regularized baselines.
  arXiv  Detail & Related papers  (2026-03-11T08:41:34Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation [72.78362530982109]
  ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation, is a framework that decouples exploration from commitment.<n>We show that naive LLM-based simulators struggle to capture rare but high-impact failure modes.<n>We introduce a risk-aware tool simulator that emphasizes fidelity on failure-inducing actions.
  arXiv  Detail & Related papers  (2026-02-02T06:33:22Z)
• Safe Langevin Soft Actor Critic [10.683491090059867]
  We introduce Safe Langevin Soft Actor-Critic (SL-SAC) to balance reward and safety in constrained reinforcement learning.<n>We show that SL-SAC achieves the lowest cost in 7 out of 10 tasks while maintaining competitive returns.<n>On Safety-Gymnasium, SL-SAC achieves cost reductions of 19-63% in velocity tasks compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-01-31T08:06:35Z)
• FoldAct: Efficient and Stable Context Folding for Long-Horizon Search Agents [6.499612808270592]
  Long-horizon reinforcement learning (RL) for large language models faces critical scalability challenges from context growth.<n>Existing approaches treat summary actions as standard actions, overlooking that summaries fundamentally modify the agent's future observation space.<n>This introduces three fundamental challenges: (1) gradient dilution where summary tokens receive insufficient training signal, (2) self-conditioning where policy updates change summary distributions, creating a vicious cycle of training collapse, and (3) computational cost from processing unique contexts at each turn.
  arXiv  Detail & Related papers  (2025-12-28T00:24:01Z)
• FR-LUX: Friction-Aware, Regime-Conditioned Policy Optimization for Implementable Portfolio Management [4.235667373386689]
  Transaction costs and regime shifts are major reasons why paper portfolios fail in live trading.<n>We introduce FR-LUX, a reinforcement learning framework that learns after-cost trading policies.<n>We provide guarantees on optimality under convex frictions, monotonic improvement under a KL trust region, long-run turnover bounds and induced inaction bands due to proportional costs.
  arXiv  Detail & Related papers  (2025-10-03T13:22:54Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts [25.205293698698867]
  We introduce Nested-ReFT, where a subset of layers of the target model acts as the behavior model to generate off-policy completions during training.<n>Our theoretical analysis shows that Nested-ReFT yields unbiased gradient estimates with controlled variance.<n>Our empirical analysis demonstrates improved computational efficiency measured as tokens/sec across multiple math reasoning benchmarks and model sizes.
  arXiv  Detail & Related papers  (2025-08-13T18:37:46Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.