ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

• URL: http://arxiv.org/abs/2608.21969v1
• Date: Sat, 22 Aug 2026 14:15:56 GMT
• Title: ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents
• Abstract summary: We propose a two-level hierarchical reinforcement learning framework for conversational agents.<n>The token-level response decoding is conditioned on the utterance-level action.<n>Our method outperforms versatile baselines in strategy determination and response quality.
• Score: 9.356868740164975
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Humans have multiple levels of temporal abstractions on daily interaction and thinking, such as concept perception and strategic planning. Inspired by this nature, we propose a two-level hierarchical reinforcement learning (RL) framework for conversational agents, bridging the gap between previous token-level or utterance-level RL methods. Developed on a two-level MDP, the token-level response decoding is conditioned on the utterance-level action, the explicit textual strategies. Based on theoretical derivation and efficiency consideration, we use DQN to solve the high-level critic and PPO to solve the low-level actor-critic. To further alleviate the reward sparsity and facilitate the convergence, we also design the dual-granularity reward mechanism, in which the utterance-level satisfaction score is integrated with token-level intrinsic motivation and K-L penalty. Experiments on both daily and emotional support conversations show that our method outperforms versatile baselines in strategy determination and response quality. Our implementation is available at https://github.com/AaronJi/ToSCA.

Related papers

• Enhancing Social Intelligence in LLMs with Hierarchical Reasoning and Utterance-Level Goal Rewarding [36.605789019423355]
  Think-Strategy-Response (TSR) framework decomposes social dialogue into two hierarchical stages: high-level strategic planning and low-level linguistic execution.<n>We introduce Linearized Hierarchical Reinforcement Learning with Variance-Gated Rewards (LHRL-VGR), a novel algorithm that dynamically routes rewards.<n> Experiments show that our approach fine-tunes a Qwen2.5-7B agent to surpass the GPT-4o baseline by 7.32% in goal completion success.
  arXiv  Detail & Related papers  (2026-08-06T10:00:50Z)
• Moira: Language-driven Hierarchical Reinforcement Learning for Pair Trading [34.923407379006626]
  Many sequential decision-making problems exhibit hierarchical structure, where high-level semantic choices constrain downstream actions and feedback is delayed and ambiguous.<n>We study this challenge through pair trading, a domain that naturally combines long-horizon semantic reasoning for asset pair selection with short-horizon execution under partial observability.<n>We formulate pair trading as a hierarchical reinforcement learning problem and propose a language-driven optimization framework in which both high-level and low-level policies are parameterized by large language models (LLMs) and optimized exclusively through prompt updates.
  arXiv  Detail & Related papers  (2026-05-03T16:37:52Z)
• Resurfacing Paralinguistic Awareness in Large Audio Language Models [64.48666302711108]
  Large Audio Language Models (LALMs) have expanded the interaction with human to speech modality.<n>LALMs usually neglect such paralinguistic cues and respond solely based on query content.<n>We propose a paralinguistic-enhanced fine-tuning (PE-FT) protocol accordingly to equip LALMs with paralinguistic-aware capabilities.
  arXiv  Detail & Related papers  (2026-03-12T13:56:42Z)
• Policy-Based Deep Reinforcement Learning Hyperheuristics for Job-Shop Scheduling Problems [3.0098885383612104]
  This paper proposes a policy-based deep reinforcement learning framework for solving the Job Shop Scheduling Problem.<n>We extend the hyper-heuristic framework with two key mechanisms.<n>We show that the proposed approach outperforms traditional deterministics, metaheuristics, and recent neural network-based scheduling methods.
  arXiv  Detail & Related papers  (2026-01-16T11:03:47Z)
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning [56.496001894673235]
  Reinforcement Learning (RL) has proven highly effective at enhancing the complex reasoning abilities of Large Language Models (LLMs)<n>Our analysis reveals that puzzling phenomena like aha moments", length-scaling'' and entropy dynamics are not disparate occurrences but hallmarks of an emergent reasoning hierarchy.
  arXiv  Detail & Related papers  (2025-09-03T18:52:49Z)
• Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation [51.06031200728449]
  We propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation.<n>Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy.<n>Results observe significant performance improvement by our method, compared with several well-known baselines.
  arXiv  Detail & Related papers  (2024-09-11T17:01:06Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoning [76.3114831562989]
  It requires Large Language Model (LLM) agents to adapt their strategies dynamically in multi-agent environments. We propose a novel framework: "K-Level Reasoning with Large Language Models (K-R)"
  arXiv  Detail & Related papers  (2024-02-02T16:07:05Z)
• Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control [44.77500987121531]
  Hierarchical Imitation Learning (HIL) has been proposed to recover highly-complex behaviors in long-horizon tasks from expert demonstrations. We develop a novel HIL algorithm based on Adversarial Inverse Reinforcement Learning. We also propose a Variational Autoencoder framework for learning with our objectives in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-10-05T00:28:26Z)
• Abstract Value Iteration for Hierarchical Reinforcement Learning [23.08652058034536]
  We propose a novel hierarchical reinforcement learning framework for control with continuous state and action spaces. A key challenge is that the ADP may not be Markov, which we address by proposing two algorithms for planning in the ADP. Our approach outperforms state-of-the-art hierarchical reinforcement learning algorithms on several challenging benchmarks.
  arXiv  Detail & Related papers  (2020-10-29T14:41:42Z)
• Tree-Structured Policy based Progressive Reinforcement Learning for Temporally Language Grounding in Video [128.08590291947544]
  Temporally language grounding in untrimmed videos is a newly-raised task in video understanding. Inspired by human's coarse-to-fine decision-making paradigm, we formulate a novel Tree-Structured Policy based Progressive Reinforcement Learning framework.
  arXiv  Detail & Related papers  (2020-01-18T15:08:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.