Moira: Language-driven Hierarchical Reinforcement Learning for Pair Trading

• URL: http://arxiv.org/abs/2605.01954v1
• Date: Sun, 03 May 2026 16:37:52 GMT
• Title: Moira: Language-driven Hierarchical Reinforcement Learning for Pair Trading
• Abstract summary: Many sequential decision-making problems exhibit hierarchical structure, where high-level semantic choices constrain downstream actions and feedback is delayed and ambiguous.<n>We study this challenge through pair trading, a domain that naturally combines long-horizon semantic reasoning for asset pair selection with short-horizon execution under partial observability.<n>We formulate pair trading as a hierarchical reinforcement learning problem and propose a language-driven optimization framework in which both high-level and low-level policies are parameterized by large language models (LLMs) and optimized exclusively through prompt updates.
• Score: 34.923407379006626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many sequential decision-making problems exhibit hierarchical structure, where high-level semantic choices constrain downstream actions and feedback is delayed and ambiguous. Learning in such settings is challenging due to credit assignment: performance degradation may arise from flawed abstractions, suboptimal execution, or their interaction. We study this challenge through pair trading, a domain that naturally combines long-horizon semantic reasoning for asset pair selection with short-horizon execution under partial observability. We formulate pair trading as a hierarchical reinforcement learning problem and propose a language-driven optimization framework in which both high-level and low-level policies are parameterized by large language models (LLMs) and optimized exclusively through prompt updates. Our approach leverages pretrained LLMs as hierarchical policies and uses trajectory- and episode-level textual feedback to adapt abstractions and execution without gradient-based fine-tuning. By explicitly separating abstraction selection from execution, the framework reduces non-stationarity across hierarchical levels and enables targeted adaptation under delayed feedback. Experiments on real-world market data show consistent improvements over traditional and LLM-based baselines, demonstrating the effectiveness of language-driven hierarchical reinforcement learning.

Related papers

• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning [80.78140312980484]
  We propose language-driven stepwise trajectory redirection, termed as STRIDE.<n>We co-train a generator and a generative verifier using only outcome-based rewards, eliminating external annotations.<n> Experiments on diverse reasoning benchmarks show that STRIDE significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-13T11:04:31Z)
• Goal-Conditioned Supervised Learning for LLM Fine-Tuning [10.43951412455864]
  We propose goal-conditioned supervised learning as an offline fine-tuning framework for large language models.<n>Our core idea is to treat feedback signals directly as an explicit goal and train the model, purely through supervised learning, to generate responses that achieve that goal.
  arXiv  Detail & Related papers  (2026-05-08T01:55:40Z)
• Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning [57.91658393667469]
  Interpretable Prompt Learning is a hybrid framework that alternates between discrete semantic token selection and continuous prompt optimization.<n>Our framework is plug-and-play, allowing seamless integration with existing prompt learning methods.
  arXiv  Detail & Related papers  (2026-05-06T02:38:59Z)
• Segment-Aligned Policy Optimization for Multi-Modal Reasoning [55.29606572822562]
  We propose a novel reinforcement learning paradigm that treats coherent reasoning steps, rather than tokens or full sequences as fundamental units of policy update.<n>Experiments on representative reasoning benchmarks demonstrate that SAPO consistently outperforms token-level and sequence-level policy optimization methods.<n>Our work underscores the importance of aligning reinforcement learning updates with the structure of reasoning, paving the way for more efficient and semantically grounded policy optimization in complex reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-02T08:47:45Z)
• Resurfacing Paralinguistic Awareness in Large Audio Language Models [64.48666302711108]
  Large Audio Language Models (LALMs) have expanded the interaction with human to speech modality.<n>LALMs usually neglect such paralinguistic cues and respond solely based on query content.<n>We propose a paralinguistic-enhanced fine-tuning (PE-FT) protocol accordingly to equip LALMs with paralinguistic-aware capabilities.
  arXiv  Detail & Related papers  (2026-03-12T13:56:42Z)
• Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigation [15.242490558864626]
  Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions and navigate through previously unseen environments.<n>We propose a retrieval-augmented framework to improve VLN without modifying or fine-tuning the underlying language model.
  arXiv  Detail & Related papers  (2026-02-17T17:00:11Z)
• Efficient RL for optimizing conversation level outcomes with an LLM-based tutor [28.21843718188098]
  Large language models (LLMs) built on existing reinforcement learning with human feedback (RLHF) frameworks typically optimize responses based on immediate turn-level human preferences.<n>We propose a method to enhance LLM-based tutors by representing the dialogue history with a lower-dimensional latent state representation of a student.<n>Our model is lightweight, requiring less computational resources than prior work of training the tutor policy end-to-end to directly output the tutor's next utterance.
  arXiv  Detail & Related papers  (2025-07-22T05:56:46Z)
• Enhancing Decision-Making of Large Language Models via Actor-Critic [28.870961806283425]
  Large Language Models (LLMs) have achieved remarkable advancements in natural language processing tasks.<n>Existing methods either rely on short-term auto-regressive action generation or face limitations in accurately simulating rollouts and assessing outcomes.<n>This paper introduces a novel LLM-based Actor-Critic framework, termed LAC, that effectively improves LLM policies with long-term action evaluations.
  arXiv  Detail & Related papers  (2025-06-04T14:58:27Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models [59.49705076369856]
  We introduce a novel framework to improve the fine-tuning phase of pre-trained language models (PLMs) We retrieve positive and negative instances from large-scale unlabeled corpora according to their domain-level and class-level semantic relatedness to a task. We then perform contrastive semi-supervised learning on both the retrieved unlabeled and original labeled instances to help PLMs capture crucial task-related semantic features.
  arXiv  Detail & Related papers  (2021-02-07T09:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.