Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents

• URL: http://arxiv.org/abs/2604.05808v2
• Date: Wed, 15 Apr 2026 02:16:31 GMT
• Title: Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents
• Abstract summary: Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks.<n>Existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability.<n>We propose STEP-HRL, a hierarchical reinforcement learning framework that enables step-level learning by conditioning only on single-step transitions.
• Score: 33.53467007340022
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks. However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability. In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories. STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task. By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress. Together, these components yield augmented step-level transitions for both high-level and low-level policies. Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage. Our code is available at https://github.com/TonyStark042/STEP-HRL.

Related papers

• Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks [0.0]
  Large Language Models (LLMs) have recently shown strong capabilities in reasoning, planning, and tool-use, enabling new forms of autonomous agents.<n>This paper introduces an LLM-Augmented Reinforcement Learning Agent that integrates LLM-driven planning with RL-based action optimization.
  arXiv  Detail & Related papers  (2026-08-04T11:44:07Z)
• SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning [21.3755929369092]
  Reinforcement Learning (RL) has emerged as a promising paradigm for training MLLM agents on dynamic GUI tasks.<n>Online RL captures the long-term dynamics but suffers from high interaction costs and potential environmental instability.<n>We propose SOLAR-RL (Semi-Online Long-horizon Assignment Reinforcement Learning), which integrates global trajectory insights directly into the offline learning process.
  arXiv  Detail & Related papers  (2026-04-24T13:53:39Z)
• SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Training [2.291770711277359]
  We propose SLEA-RL (Step-Level Experience-Augmented Reinforcement Learning), a framework that retrieves relevant experiences at each decision step conditioned on the current observation.<n>SLEA-RL operates through three components: (i) step-level observation clustering that groups structurally equivalent environmental states for efficient cluster-indexed retrieval; (ii) a self-evolving experience library that distills successful strategies and failure patterns through score-based admission and rate-limited extraction; and (iii) policy optimization with step-level credit assignment for fine-grained advantage estimation across multi-turn episodes.
  arXiv  Detail & Related papers  (2026-03-18T07:16:18Z)
• Sample-Efficient Neurosymbolic Deep Reinforcement Learning [49.60927398960061]
  We propose a neuro-symbolic Deep RL approach that integrates background symbolic knowledge to improve sample efficiency.<n>Online reasoning is performed to guide the training process through two mechanisms.<n>We show improved performance over a state-of-the-art reward machine baseline.
  arXiv  Detail & Related papers  (2026-01-06T09:28:53Z)
• DRL: Discriminative Representation Learning with Parallel Adapters for Class Incremental Learning [63.65467569295623]
  We propose the Discriminative Representation Learning (DRL) framework to specifically address these challenges.<n>To conduct incremental learning effectively and yet efficiently, the DRL's network is built upon a PTM.<n>Our DRL consistently outperforms other state-of-the-art methods throughout the entire CIL period.
  arXiv  Detail & Related papers  (2025-10-14T03:19:15Z)
• PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness [57.020401590532686]
  We introduce PARL-MT, a framework that explicitly incorporates progress awareness into LLM training for multi-turn function calling.<n>We show that PARL-MT significantly outperforms existing methods, highlighting the effectiveness of progress awareness in enabling robust and efficient multi-turn function calling.
  arXiv  Detail & Related papers  (2025-09-27T09:32:27Z)
• Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning [32.260964481673085]
  Large language models (LLMs) struggle with long-horizon decision-making tasks due to deficient exploration and long-term credit assignment.<n>We propose an innovative framework that introduces a parameter-efficient and generally applicable hierarchy to LLM policies.<n>We develop a scheme where the low-level controller is supervised with abstract, step-by-step plans that are learned and instructed by the high-level policy.
  arXiv  Detail & Related papers  (2025-05-26T09:43:40Z)
• Boosting Hierarchical Reinforcement Learning with Meta-Learning for Complex Task Adaptation [0.0]
  Hierarchical Reinforcement Learning (HRL) is well-suited for solving complex tasks by breaking them down into structured policies.<n>We propose integrating meta-learning into HRL to enable agents to learn and adapt hierarchical policies more effectively.
  arXiv  Detail & Related papers  (2024-10-10T13:47:37Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Multi-granularity Knowledge Transfer for Continual Reinforcement Learning [10.89257691485739]
  Continual reinforcement learning (CRL) empowers RL agents with the ability to learn a sequence of tasks.<n>Existing methods often focus on transferring fine-grained knowledge across similar tasks.<n>We propose a novel framework called MT-Core to enhance coarse-grained knowledge transfer.
  arXiv  Detail & Related papers  (2024-01-25T03:06:51Z)
• BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization [34.24884427152513]
  We propose a general meta ERL framework via bilevel optimization (BiERL) We design an elegant meta-level architecture that embeds the inner-level's evolving experience into an informative population representation. We perform extensive experiments in MuJoCo and Box2D tasks to verify that as a general framework, BiERL outperforms various baselines and consistently improves the learning performance for a diversity of ERL algorithms.
  arXiv  Detail & Related papers  (2023-08-01T09:31:51Z)
• Provable Hierarchy-Based Meta-Reinforcement Learning [50.17896588738377]
  We analyze HRL in the meta-RL setting, where learner learns latent hierarchical structure during meta-training for use in a downstream task. We provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.
  arXiv  Detail & Related papers  (2021-10-18T17:56:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.