What Teaches Robots to Walk, Teaches Them to Trade too -- Regime Adaptive Execution using Informed Data and LLMs

• URL: http://arxiv.org/abs/2406.15508v1
• Date: Thu, 20 Jun 2024 00:17:28 GMT
• Title: What Teaches Robots to Walk, Teaches Them to Trade too -- Regime Adaptive Execution using Informed Data and LLMs
• Authors: Raeid Saqur,
• Abstract summary: We introduce an innovative approach that leverages world knowledge of pretrained LLMs (aka. 'privileged information' in robotics) and dynamically adapts them using intrinsic, natural market rewards. Strong empirical results demonstrate the efficacy of our method in adapting to regime shifts in financial markets. The proposed algorithmic framework outperforms best-performing SOTA LLM models on the existing (FLARE) benchmark stock-movement (SM) tasks by more than 15% improved accuracy.
• Score: 1.6317061277457001
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Machine learning techniques applied to the problem of financial market forecasting struggle with dynamic regime switching, or underlying correlation and covariance shifts in true (hidden) market variables. Drawing inspiration from the success of reinforcement learning in robotics, particularly in agile locomotion adaptation of quadruped robots to unseen terrains, we introduce an innovative approach that leverages world knowledge of pretrained LLMs (aka. 'privileged information' in robotics) and dynamically adapts them using intrinsic, natural market rewards using LLM alignment technique we dub as "Reinforcement Learning from Market Feedback" (**RLMF**). Strong empirical results demonstrate the efficacy of our method in adapting to regime shifts in financial markets, a challenge that has long plagued predictive models in this domain. The proposed algorithmic framework outperforms best-performing SOTA LLM models on the existing (FLARE) benchmark stock-movement (SM) tasks by more than 15\% improved accuracy. On the recently proposed NIFTY SM task, our adaptive policy outperforms the SOTA best performing trillion parameter models like GPT-4. The paper details the dual-phase, teacher-student architecture and implementation of our model, the empirical results obtained, and an analysis of the role of language embeddings in terms of Information Gain.

Related papers

• Scaling Autonomous Agents via Automatic Reward Modeling And Planning [52.39395405893965]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks. However, they still struggle with problems requiring multi-step decision-making and environmental feedback. We propose a framework that can automatically learn a reward model from the environment without human annotations.
  arXiv  Detail & Related papers  (2025-02-17T18:49:25Z)
• From Selection to Generation: A Survey of LLM-based Active Learning [153.8110509961261]
  Large Language Models (LLMs) have been employed for generating entirely new data instances and providing more cost-effective annotations. This survey aims to serve as an up-to-date resource for researchers and practitioners seeking to gain an intuitive understanding of LLM-based AL techniques.
  arXiv  Detail & Related papers  (2025-02-17T12:58:17Z)
• Improve LLM-as-a-Judge Ability as a General Ability [40.2210529561692]
  Large language models (LLMs) can evaluate responses across diverse scenarios, providing accurate preference signals. Recent studies have raised many methods to train LLM as generative judges, but most of them are data consuming or lack accuracy. In this work, we implement a two-stage training approach, comprising supervised fine-tuning (SFT) warm-up and direct preference optimization (DPO) enhancement.
  arXiv  Detail & Related papers  (2025-02-17T11:28:43Z)
• Reinforcement-Learning Portfolio Allocation with Dynamic Embedding of Market Information [12.032301674764552]
  We develop a portfolio allocation framework that leverages deep learning techniques to address challenges arising from high-dimensional, non-stationary, and low-signal-to-noise market information. We design a reinforcement learning framework that integrates generative autoencoders and online meta-learning to dynamically embed market information. Empirical analysis based on the top 500 U.S. stocks demonstrates that our framework outperforms common portfolio benchmarks.
  arXiv  Detail & Related papers  (2025-01-29T20:56:59Z)
• Exploratory Mean-Variance Portfolio Optimization with Regime-Switching Market Dynamics [3.6149777601911097]
  We study a regime-switching market setting and apply reinforcement learning techniques to assist informed exploration within the control space. In a real market data study, EMVRS with OC learning outperforms its counterparts with the highest mean and reasonably low volatility of the annualized portfolio returns.
  arXiv  Detail & Related papers  (2025-01-28T02:48:41Z)
• Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment [65.15914284008973]
  We propose to leverage an Inverse Reinforcement Learning (IRL) technique to simultaneously build an reward model and a policy model. We show that the proposed algorithms converge to the stationary solutions of the IRL problem. Our results indicate that it is beneficial to leverage reward learning throughout the entire alignment process.
  arXiv  Detail & Related papers  (2024-05-28T07:11:05Z)
• ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling [15.67321902882617]
  We introduce OR-Instruct, a semi-automated data synthesis framework for optimization modeling. We also introduce IndustryOR, the first industrial benchmark for evaluating LLMs in solving practical OR problems.
  arXiv  Detail & Related papers  (2024-05-28T01:55:35Z)
• Learning Objective-Specific Active Learning Strategies with Attentive Neural Processes [72.75421975804132]
  Learning Active Learning (LAL) suggests to learn the active learning strategy itself, allowing it to adapt to the given setting. We propose a novel LAL method for classification that exploits symmetry and independence properties of the active learning problem. Our approach is based on learning from a myopic oracle, which gives our model the ability to adapt to non-standard objectives.
  arXiv  Detail & Related papers  (2023-09-11T14:16:37Z)
• Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models [51.3422222472898]
  We document the capability of large language models (LLMs) like ChatGPT to predict stock price movements using news headlines. We develop a theoretical model incorporating information capacity constraints, underreaction, limits-to-arbitrage, and LLMs.
  arXiv  Detail & Related papers  (2023-04-15T19:22:37Z)
• Reinforced Deep Markov Models With Applications in Automatic Trading [0.0]
  We propose a model-based RL approach, coined Reinforced Deep Markov Model (RDMM) RDMM integrates desirable properties of a reinforcement learning algorithm acting as an automatic trading system. Tests show that the RDMM is data-efficient and provides financial gains compared to the benchmarks in the optimal execution problem.
  arXiv  Detail & Related papers  (2020-11-09T12:46:30Z)
• Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources [78.72922528736011]
  We propose a novel approach, black-box adversarial reprogramming (BAR), that repurposes a well-trained black-box machine learning model. Using zeroth order optimization and multi-label mapping techniques, BAR can reprogram a black-box ML model solely based on its input-output responses. BAR outperforms state-of-the-art methods and yields comparable performance to the vanilla adversarial reprogramming method.
  arXiv  Detail & Related papers  (2020-07-17T01:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.