Empathy Level Alignment via Reinforcement Learning for Empathetic Response Generation

• URL: http://arxiv.org/abs/2408.02976v3
• Date: Sun, 02 Mar 2025 08:30:58 GMT
• Title: Empathy Level Alignment via Reinforcement Learning for Empathetic Response Generation
• Authors: Hui Ma, Bo Zhang, Bo Xu, Jian Wang, Hongfei Lin, Xiao Sun,
• Abstract summary: We propose an empathetic response generation framework using reinforcement learning (EmpRL)<n>The framework develops an effective empathy reward function and generates empathetic responses by maximizing the expected reward through reinforcement learning.<n>Both automatic and human evaluations demonstrate that the proposed EmpRL framework significantly improves the quality of generated responses.
• Score: 17.210469986755747
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Empathetic response generation, aiming to understand the user's situation and feelings and respond empathically, is crucial in building human-like dialogue systems. Traditional approaches typically employ maximum likelihood estimation as the optimization objective during training, yet fail to align the empathy levels between generated and target responses. To this end, we propose an empathetic response generation framework using reinforcement learning (EmpRL). The framework develops an effective empathy reward function and generates empathetic responses by maximizing the expected reward through reinforcement learning. EmpRL utilizes the pre-trained T5 model as the generator and further fine-tunes it to initialize the policy. To align the empathy levels between generated and target responses within a given context, an empathy reward function containing three empathy communication mechanisms -- emotional reaction, interpretation, and exploration -- is constructed using pre-designed and pre-trained empathy identifiers. During reinforcement learning training, the proximal policy optimization algorithm is used to fine-tune the policy, enabling the generation of empathetic responses. Both automatic and human evaluations demonstrate that the proposed EmpRL framework significantly improves the quality of generated responses, enhances the similarity in empathy levels between generated and target responses, and produces empathetic responses covering both affective and cognitive aspects.

Related papers

• Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialogue [53.95386201009769]
  We introduce EmpathyEval, a descriptive natural-language-based evaluation model for assessing empathetic quality in spoken dialogues.<n>We propose ReEmpathy, an end-to-end Spoken Language Models that enhances empathetic dialogue through a novel Empathetic Self-Reflective Alternating Inference mechanism.
  arXiv  Detail & Related papers  (2026-01-26T09:04:50Z)
• Cause-Aware Empathetic Response Generation via Chain-of-Thought Fine-Tuning [12.766893968788263]
  Empathetic response generation endows agents with the capability to comprehend dialogue contexts and react to expressed emotions. Previous works predominantly focus on leveraging the speaker's emotional labels, but ignore the importance of emotion cause reasoning. We propose a cause-aware empathetic generation approach by integrating emotions and causes through a well-designed Chain-of-Thought prompt.
  arXiv  Detail & Related papers  (2024-08-21T13:11:03Z)
• APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation [71.26755736617478]
  Empathetic response generation is designed to comprehend the emotions of others. We develop a framework that combines retrieval augmentation and emotional support strategy integration. Our framework can enhance the empathy ability of LLMs from both cognitive and affective empathy perspectives.
  arXiv  Detail & Related papers  (2024-07-23T02:23:37Z)
• EmPO: Emotion Grounding for Empathetic Response Generation through Preference Optimization [9.934277461349696]
  Empathetic response generation is a desirable aspect of conversational agents. We propose a novel approach where we construct theory-driven preference datasets based on emotion grounding. We show that LLMs can be aligned for empathetic response generation by preference optimization while retaining their general performance.
  arXiv  Detail & Related papers  (2024-06-27T10:41:22Z)
• Improving Empathetic Dialogue Generation by Dynamically Infusing Commonsense Knowledge [39.536604198392375]
  In empathetic conversations, individuals express their empathy towards others. Previous work has mainly focused on generating empathetic responses by utilizing the speaker's emotion. We propose a novel approach for empathetic response generation, which incorporates an adaptive module for commonsense knowledge selection.
  arXiv  Detail & Related papers  (2023-05-24T10:25:12Z)
• Use of a Taxonomy of Empathetic Response Intents to Control and Interpret Empathy in Neural Chatbots [4.264192013842096]
  A recent trend in the domain of open-domain conversational agents is enabling them to converse empathetically to emotional prompts. Current approaches either follow an end-to-end approach or condition the responses on similar emotion labels to generate empathetic responses. We propose several rule-based and neural approaches to predict the next response's emotion/intent and generate responses conditioned on these predicted emotions/intents.
  arXiv  Detail & Related papers  (2023-05-17T10:03:03Z)
• CASE: Aligning Coarse-to-Fine Cognition and Affection for Empathetic Response Generation [59.8935454665427]
  Empathetic dialogue models usually consider only the affective aspect or treat cognition and affection in isolation. We propose the CASE model for empathetic dialogue generation.
  arXiv  Detail & Related papers  (2022-08-18T14:28:38Z)
• EmpBot: A T5-based Empathetic Chatbot focusing on Sentiments [75.11753644302385]
  Empathetic conversational agents should not only understand what is being discussed, but also acknowledge the implied feelings of the conversation partner. We propose a method based on a transformer pretrained language model (T5) We evaluate our model on the EmpatheticDialogues dataset using both automated metrics and human evaluation.
  arXiv  Detail & Related papers  (2021-10-30T19:04:48Z)
• CEM: Commonsense-aware Empathetic Response Generation [31.956147246779423]
  We propose a novel approach for empathetic response generation, which leverages commonsense to draw more information about the user's situation. We evaluate our approach on EmpatheticDialogues, which is a widely-used benchmark dataset for empathetic response generation.
  arXiv  Detail & Related papers  (2021-09-13T06:55:14Z)
• Exemplars-guided Empathetic Response Generation Controlled by the Elements of Human Communication [88.52901763928045]
  We propose an approach that relies on exemplars to cue the generative model on fine stylistic properties that signal empathy to the interlocutor. We empirically show that these approaches yield significant improvements in empathetic response quality in terms of both automated and human-evaluated metrics.
  arXiv  Detail & Related papers  (2021-06-22T14:02:33Z)
• Emotion-aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction [55.47134146639492]
  This article proposes a unifed end-to-end neural architecture, which is capable of simultaneously encoding the semantics and the emotions in a post. Experiments on real-world data demonstrate that the proposed method outperforms the state-of-the-art methods in terms of both content coherence and emotion appropriateness.
  arXiv  Detail & Related papers  (2021-06-06T06:26:15Z)
• Improving Response Quality with Backward Reasoning in Open-domain Dialogue Systems [53.160025961101354]
  We propose to train the generation model in a bidirectional manner by adding a backward reasoning step to the vanilla encoder-decoder training. The proposed backward reasoning step pushes the model to produce more informative and coherent content. Our method can improve response quality without introducing side information.
  arXiv  Detail & Related papers  (2021-04-30T20:38:27Z)
• Counterfactual Off-Policy Training for Neural Response Generation [94.76649147381232]
  We propose to explore potential responses by counterfactual reasoning. Training on the counterfactual responses under the adversarial learning framework helps to explore the high-reward area of the potential response space. An empirical study on the DailyDialog dataset shows that our approach significantly outperforms the HRED model.
  arXiv  Detail & Related papers  (2020-04-29T22:46:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.