DecoupledESC: Enhancing Emotional Support Generation via Strategy-Response Decoupled Preference Optimization

• URL: http://arxiv.org/abs/2505.16995v1
• Date: Thu, 22 May 2025 17:56:21 GMT
• Title: DecoupledESC: Enhancing Emotional Support Generation via Strategy-Response Decoupled Preference Optimization
• Authors: Chao Zhang, Xin Shi, Xueqiao Zhang, Yifan Zhu, Yi Yang, Yawei Luo,
• Abstract summary: We propose a Decoupled ESC framework inspired by Gross's Extended Process Model of Emotion Regulation.<n>Our framework outperforms joint optimization baselines, reducing preference bias and improving response quality.
• Score: 35.50223358356217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in Emotional Support Conversation (ESC) have improved emotional support generation by fine-tuning Large Language Models (LLMs) via Supervised Fine-Tuning (SFT). However, common psychological errors still persist. While Direct Preference Optimization (DPO) shows promise in reducing such errors through pairwise preference learning, its effectiveness in ESC tasks is limited by two key challenges: (1) Entangled data structure: Existing ESC data inherently entangles psychological strategies and response content, making it difficult to construct high-quality preference pairs; and (2) Optimization ambiguity: Applying vanilla DPO to such entangled pairwise data leads to ambiguous training objectives. To address these issues, we introduce Inferential Preference Mining (IPM) to construct high-quality preference data, forming the IPM-PrefDial dataset. Building upon this data, we propose a Decoupled ESC framework inspired by Gross's Extended Process Model of Emotion Regulation, which decomposes the ESC task into two sequential subtasks: strategy planning and empathic response generation. Each was trained via SFT and subsequently enhanced by DPO to align with the psychological preference. Extensive experiments demonstrate that our Decoupled ESC framework outperforms joint optimization baselines, reducing preference bias and improving response quality.

Related papers

• 2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization [3.674552982566341]
  2D-Curri-DPO is a novel framework employing a two-dimensional curriculum that jointly models Prompt Complexity (PC) and Pairwise Distinguishability.<n>Our approach achieves state-of-the-art performance on challenging test sets like UltraFeedback.
  arXiv  Detail & Related papers  (2025-04-10T15:32:00Z)
• Chain of Strategy Optimization Makes Large Language Models Better Emotional Supporter [44.17098675825127]
  We propose Chain-of-Strategy Optimization (CSO), a novel approach that optimize strategy selection preferences at each dialogue turn.<n>We first leverage Monte Carlo Tree Search to construct ESC-Pro, a high-quality preference dataset with turn-level strategy-response pairs.<n>Training on ESC-Pro with CSO improves both strategy accuracy and bias mitigation, enabling LLMs to generate more empathetic and contextually appropriate responses.
  arXiv  Detail & Related papers  (2025-03-07T12:07:59Z)
• PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation [5.347428263669927]
  Post-Training Extrapolation Optimization (PEO) is a novel and efficient framework for bi-factorial alignment.<n>PEO generates a family of optimal policies in a single training pass by leveraging a three-phase pipeline.
  arXiv  Detail & Related papers  (2025-03-03T06:56:39Z)
• A Systematic Examination of Preference Learning through the Lens of Instruction-Following [83.71180850955679]
  We use a novel synthetic data generation pipeline to generate 48,000 instruction unique-following prompts.<n>With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS)<n>Experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements.<n>High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance.
  arXiv  Detail & Related papers  (2024-12-18T15:38:39Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness [27.43137305486112]
  We propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-09-26T12:37:26Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Adaptive Preference Scaling for Reinforcement Learning with Human Feedback [103.36048042664768]
  Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values. We propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO) Our method is versatile and can be readily adapted to various preference optimization frameworks.
  arXiv  Detail & Related papers  (2024-06-04T20:33:22Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.