Towards Open-Ended Emotional Support Conversations in LLMs via Reinforcement Learning with Future-Oriented Rewards

• URL: http://arxiv.org/abs/2508.12935v1
• Date: Mon, 18 Aug 2025 14:04:26 GMT
• Title: Towards Open-Ended Emotional Support Conversations in LLMs via Reinforcement Learning with Future-Oriented Rewards
• Abstract summary: Emotional Support Conversation systems aim to alleviate users' emotional difficulties and provide long-term, systematic support for emotional well-being.<n>Most large language model (LLM)-based ESC systems rely on predefined strategies, which limits their effectiveness in complex, real-life scenarios.<n>This paper introduces a novel end-to-end framework (RLFF-ESC) that directly learns enduring emotionally supportive response skills using reinforcement learning.
• Score: 13.938394655357916
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Emotional Support Conversation (ESC) systems aim to alleviate users' emotional difficulties and provide long-term, systematic support for emotional well-being. However, most large language model (LLM)-based ESC systems rely on predefined strategies, which limits their effectiveness in complex, real-life scenarios. To enable flexible responses to diverse emotional problem scenarios, this paper introduces a novel end-to-end framework (RLFF-ESC) that directly learns enduring emotionally supportive response skills using reinforcement learning. For sustained emotional support, we first employ an LLM-based multi-agent mechanism to simulate future dialogue trajectories and collect future-oriented rewards. We then train a future-oriented reward model, which is subsequently used to train the emotional support policy model. Additionally, we incorporate an explicit reasoning process during response generation to further enhance the quality, relevance, and contextual appropriateness of the system's responses. We evaluate the backbone policy model on Qwen2.5-7B-Instruct-1M and LLaMA3.1-8B-Instruct models, testing the proposed RLFF-ESC framework across two public ESC datasets. Experimental results demonstrate that RLFF-ESC consistently outperforms existing baselines in terms of goal completion and response quality.

Related papers

• ESC-Skills: Discovering and Self-Evolving Skills for Emotional Support Conversations [25.708317827505695]
  We propose ESC-Skills, a skill-centric framework that discovers and self-evolves executable emotional support skills.<n>We first model localized support interactions as Intervention Units (IUs), which capture state--action-outcome dynamics between seeker states, support interventions, and post-response emotional changes.<n>We construct the ESC-Skills Bank, a repository of executable emotional support skills containing intervention guidance, applicability conditions, expected outcomes, and potential risks.
  arXiv  Detail & Related papers  (2026-05-27T03:32:12Z)
• EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models [62.3977734456669]
  We propose Reflective Reinforcement Learning for Emotional Reasoning (EMO-R3), a framework designed to enhance the emotional reasoning ability of Multimodal Large Language Models (MLLMs)<n>We introduce Structured Emotional Thinking to guide the model to perform step-by-step emotional reasoning in a structured and interpretable manner, and design a Reflective Emotional Reward that enables the model to re-evaluate its reasoning based on visual-text consistency and emotional coherence.<n>EMO-R3 significantly improves both the interpretability and emotional intelligence of MLLMs, achieving superior performance across multiple visual emotional understanding benchmarks.
  arXiv  Detail & Related papers  (2026-02-27T08:42:52Z)
• Emotional Support with LLM-based Empathetic Dialogue Generation [5.289702620838033]
  This paper presents our solution for the NLPCC 2025 Task 8 ESC evaluation.<n>We leverage large-scale language models enhanced by prompt engineering and finetuning techniques.
  arXiv  Detail & Related papers  (2025-07-17T06:24:20Z)
• IntentionESC: An Intention-Centered Framework for Enhancing Emotional Support in Dialogue Systems [74.0855067343594]
  In emotional support conversations, unclear intentions can lead supporters to employ inappropriate strategies.<n>We propose the Intention-centered Emotional Support Conversation framework.<n>It defines the possible intentions of supporters, identifies key emotional state aspects for inferring these intentions, and maps them to appropriate support strategies.
  arXiv  Detail & Related papers  (2025-06-06T10:14:49Z)
• Convert Language Model into a Value-based Strategic Planner [11.070654717643816]
  Emotional support conversation (ESC) aims to alleviate the emotional distress of individuals through effective conversations.<n>We propose a framework called straQ* to define the diagram from the state model perspective.<n>Our framework allows a plug-and-play LLM to bootstrap the planning during ESC, determine the optimal strategy based on long-term returns, and finally guide the LLM to response.
  arXiv  Detail & Related papers  (2025-05-11T14:13:58Z)
• FiSMiness: A Finite State Machine Based Paradigm for Emotional Support Conversations [11.718316719735832]
  Emotional support conversation (ESC) aims to alleviate the emotional distress of individuals through effective conversations.<n>We leverage the Finite State Machine (FSM) on large language models and propose a framework called FiSMiness.<n>Our framework allows a single LLM to bootstrap the planning during ESC, and self-reason the seeker's emotion, support strategy and the final response upon each conversational turn.
  arXiv  Detail & Related papers  (2025-04-16T07:52:06Z)
• Chain of Strategy Optimization Makes Large Language Models Better Emotional Supporter [71.17800993332175]
  We propose Chain-of-Strategy Optimization (CSO), a novel approach that optimize strategy selection preferences at each dialogue turn.<n>We first leverage Monte Carlo Tree Search to construct ESC-Pro, a high-quality preference dataset with turn-level strategy-response pairs.<n>Training on ESC-Pro with CSO improves both strategy accuracy and bias mitigation, enabling LLMs to generate more empathetic and contextually appropriate responses.
  arXiv  Detail & Related papers  (2025-03-07T12:07:59Z)
• ESC-Eval: Evaluating Emotion Support Conversations in Large Language Models [55.301188787490545]
  Emotion Support Conversation (ESC) aims to reduce human stress, offer emotional guidance, and enhance human mental and physical well-being. We propose an ESC Evaluation framework (ESC-Eval), which uses a role-playing agent to interact with ESC models. We conduct comprehensive human annotations on interactive multi-turn dialogues of different ESC models.
  arXiv  Detail & Related papers  (2024-06-21T08:03:33Z)
• FEEL: A Framework for Evaluating Emotional Support Capability with Large Language Models [14.894922829587841]
  Emotional Support Conversation (ESC) is a typical dialogue that can effectively assist the user in mitigating emotional pressures. Current non-artificial methodologies face challenges in effectively appraising the emotional support capability. We propose a novel model FEEL, employing Large Language Models (LLMs) as evaluators to assess emotional support capabilities.
  arXiv  Detail & Related papers  (2024-03-23T03:32:26Z)
• Building Emotional Support Chatbots in the Era of LLMs [64.06811786616471]
  We introduce an innovative methodology that synthesizes human insights with the computational prowess of Large Language Models (LLMs) By utilizing the in-context learning potential of ChatGPT, we generate an ExTensible Emotional Support dialogue dataset, named ExTES. Following this, we deploy advanced tuning techniques on the LLaMA model, examining the impact of diverse training strategies, ultimately yielding an LLM meticulously optimized for emotional support interactions.
  arXiv  Detail & Related papers  (2023-08-17T10:49:18Z)
• Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planning [81.79431311952656]
  We propose a novel system MultiESC to provide Emotional Support. For strategy planning, we propose lookaheads to estimate the future user feedback after using particular strategies. For user state modeling, MultiESC focuses on capturing users' subtle emotional expressions and understanding their emotion causes.
  arXiv  Detail & Related papers  (2022-10-09T12:23:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.