Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance

• URL: http://arxiv.org/abs/2606.22350v1
• Date: Sun, 21 Jun 2026 06:15:42 GMT
• Title: Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance
• Abstract summary: Reinforcement Learning (RL) has been widely applied to sequential decisionmaking, yet it often suffers from poor sample efficiency due to costly interactions with the environment.<n>We propose emphHierarchical Reinforcement Learning with Language Instructions (HRLLI), a hierarchical RL framework that explicitly models natural-language instructions as dynamically selectable semantic guidance during decisionmaking.
• Score: 21.023547622557775
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) has been widely applied to sequential decision-making, yet it often suffers from poor sample efficiency due to costly interactions with the environment. A limited line of recent work has started exploring improving RL efficiency by leveraging external knowledge expressed in natural-language instructions. However, the few existing approaches typically treat the entire instruction as a single conditioning input, failing to account for the stage-dependent nature of language guidance, especially in complex environments. In this paper, we propose \emph{Hierarchical Reinforcement Learning with Language Instructions (HRLLI)}, a hierarchical RL framework that explicitly models natural-language instructions as dynamically selectable semantic guidance during decision-making. HRLLI decomposes instructions into a set of piecewise guidance elements, where each instruction piece may become relevant at different stages of interaction with the environment. A novel hierarchical RL policy structure is then formulated in a \emph{Select-to-Act} paradigm: a high-level semantic policy acts as a guidance selector that selects the most relevant instruction piece to the current state to guide the low-level agent's decision, while a low-level policy executes environment actions conditioned on the selected guidance. The two-level policies are learned simultaneously to maximize augmented expected returns from interactions with the environment. This design enables the agent to adaptively ground language instructions into stage-specific decisions during interaction. Experiments on the instruction-intensive RTFM benchmark show that HRLLI consistently outperforms strong instruction-conditioned RL baselines, demonstrating that explicitly modeling adaptive instruction selection significantly improves the effectiveness of RL.

Related papers

• Learning to Route Languages for Multilingual Policy Optimization [50.044784120224335]
  Large language models are trained on heterogeneous multilingual corpora.<n>We propose language-routed policy optimization (LRPO), an online policy optimization framework that treats language as a selectable variable.
  arXiv  Detail & Related papers  (2026-05-25T02:28:41Z)
• Moira: Language-driven Hierarchical Reinforcement Learning for Pair Trading [34.923407379006626]
  Many sequential decision-making problems exhibit hierarchical structure, where high-level semantic choices constrain downstream actions and feedback is delayed and ambiguous.<n>We study this challenge through pair trading, a domain that naturally combines long-horizon semantic reasoning for asset pair selection with short-horizon execution under partial observability.<n>We formulate pair trading as a hierarchical reinforcement learning problem and propose a language-driven optimization framework in which both high-level and low-level policies are parameterized by large language models (LLMs) and optimized exclusively through prompt updates.
  arXiv  Detail & Related papers  (2026-05-03T16:37:52Z)
• Self-Guided Plan Extraction for Instruction-Following Tasks with Goal-Conditional Reinforcement Learning [49.0945829107309]
  SuperIgor enables a language model to generate and refine high-level plans through a self-learning mechanism.<n>Our approach involves iterative co-training: an RL agent is trained to follow the generated plans, while the language model adapts and modifies these plans based on RL feedback and preferences.<n>Results show that SuperIgor agents adhere to instructions more strictly than baseline methods, while also demonstrating strong generalization to previously unseen instructions.
  arXiv  Detail & Related papers  (2026-04-22T14:19:23Z)
• Zero-Shot Instruction Following in RL via Structured LTL Representations [54.08661695738909]
  Linear temporal logic (LTL) is a compelling framework for specifying complex, structured tasks for reinforcement learning (RL) agents.<n>Recent work has shown that interpreting instructions as finite automata, which can be seen as high-level programs monitoring task progress, enables learning a single generalist policy capable of executing arbitrary instructions at test time.<n>We propose a novel approach to learning a multi-task policy for following arbitrary instructions that addresses this shortcoming.
  arXiv  Detail & Related papers  (2025-12-02T10:44:51Z)
• Agentic Policy Optimization via Instruction-Policy Co-Evolution [44.74237684380034]
  INSPO is a novel framework for instruction-policy co-evolution.<n>It integrates instruction optimization as a dynamic component of the reinforcement learning loop.<n>In experiments, INSPO achieves substantial performance gains with only a marginal increase in computational overhead.
  arXiv  Detail & Related papers  (2025-12-01T17:56:29Z)
• Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Experts [63.412646471177645]
  We propose a novel Reinforced Strategy Optimization (RSO) method for Conversational Recommender Systems (CRSs)<n>RSO decomposes the process of generating strategy-driven response decisions into the macro-level strategy planning and micro-level strategy adaptation.<n>Experiments show that RSO significantly improves interaction performance compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-09-30T11:12:01Z)
• A Fuzzy Logic Prompting Framework for Large Language Models in Adaptive and Uncertain Tasks [2.1756081703276]
  We introduce a modular prompting framework that supports safer and more adaptive use of large language models (LLMs) across dynamic, user-centered tasks.<n>Our method combines a natural language boundary prompt with a control schema encoded with fuzzy scaffolding logic and adaptation rules.<n>In a simulated intelligent tutoring setting, the framework improves scaffolding quality, adaptivity, and instructional alignment across multiple models, outperforming standard prompting baselines.
  arXiv  Detail & Related papers  (2025-08-08T23:50:48Z)
• Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments [42.06453257292203]
  We propose a hierarchical framework that combines the deep language comprehension of large language models with the adaptive action-execution capabilities of reinforcement learning agents. We have demonstrated the effectiveness of our approach in two different environments: in IGLU, where agents are instructed to build structures, and in Crafter, where agents perform tasks and interact with objects in the surrounding environment according to language commands.
  arXiv  Detail & Related papers  (2024-07-12T14:19:36Z)
• AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents [74.17623527375241]
  We introduce a novel framework, called AutoGuide, which automatically generates context-aware guidelines from offline experiences.<n>As a result, our guidelines facilitate the provision of relevant knowledge for the agent's current decision-making process.<n>Our evaluation demonstrates that AutoGuide significantly outperforms competitive baselines in complex benchmark domains.
  arXiv  Detail & Related papers  (2024-03-13T22:06:03Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• ELLA: Exploration through Learned Language Abstraction [6.809870486883877]
  ELLA is a reward shaping approach that correlates high-level instructions with simpler low-level instructions to enrich the sparse rewards afforded by the environment. ELLA shows a significant gain in sample efficiency across several environments compared to competitive language-based reward shaping and no-shaping methods.
  arXiv  Detail & Related papers  (2021-03-10T02:18:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.