Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning

• URL: http://arxiv.org/abs/2603.15434v2
• Date: Wed, 18 Mar 2026 08:59:23 GMT
• Title: Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning
• Abstract summary: We propose Reaction Aware Policy optimization (RAPO) for emotional support systems.<n>RAPO treats dialogue as a reaction-driven process and utilizes simulated user responses to generate dense natural-language feedback.<n>RAPO significantly outperforms strong reinforcement learning baselines in driving positive interaction outcomes.
• Score: 42.070531160001785
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While current emotional support dialogue systems typically rely on expert-defined scalar rewards for alignment, these signals suffer from severe information sparsity. They cannot explain why a response failed or how to adapt to dynamic user states, often diverging from the actual goal of facilitating positive emotional shifts. In practice, the most direct and reliable learning signal emerges from the user's continuous reactions during ongoing interaction. We therefore propose Reaction Aware Policy Optimization (RAPO), a framework that optimizes over interaction consequences rather than rubric scores. RAPO treats dialogue as a reaction-driven process and utilizes simulated user responses to generate dense natural-language feedback through three core components: Hindsight Dialogue Selection, which isolates pivotal turns that meaningfully alter user emotional trajectories; Generative Hindsight Feedback, which transforms user reactions into contrastive ranking signals and natural-language critiques; and Scalar-Verbal Hybrid Policy Optimization, which couples scalar reward optimization for global alignment with verbal feedback distillation for fine-grained semantic refinement. Extensive experiments on ESC and Sotopia demonstrate that RAPO significantly outperforms strong reinforcement learning baselines in driving positive interaction outcomes.

Related papers

• REFINE: Real-world Exploration of Interactive Feedback and Student Behaviour [12.059739840538638]
  We introduce REFINE, a locally deployable, multi-agent feedback system that treats feedback as an interactive process.<n> REFINE combines a pedagogically-grounded feedback generation agent with an LLM-as-a-judge-guided regeneration loop using a human-aligned judge.<n>Our findings demonstrate the feasibility and effectiveness of multi-agent, tool-augmented feedback systems for scalable, interactive feedback.
  arXiv  Detail & Related papers  (2026-03-31T01:48:08Z)
• GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization [5.5806589487812746]
  GDPO-Listener is a novel framework that achieves highly expressive speaking and listening motion generation.<n>We introduce an Auto-Regressive Flow Matching architecture enabling stable supervised learning.<n>By isolating reward normalization across distinct FLAME parameter groups, GDPO explicitly incentivizes high variance expressive generations.
  arXiv  Detail & Related papers  (2026-03-26T04:36:27Z)
• EmoLLM: Appraisal-Grounded Cognitive-Emotional Co-Reasoning in Large Language Models [15.744777556347708]
  EmoLLM is an appraisal-grounded framework for IQ/EQ co-reasoning in dialogue.<n>We train EmoLLM in a multi-turn role-play environment with reinforcement learning.<n>EmoLLM improves emotional state outcomes and response quality over strong baselines.
  arXiv  Detail & Related papers  (2026-03-17T14:17:50Z)
• ReactMotion: Generating Reactive Listener Motions from Speaker Utterance [93.89117265072464]
  Reactive Listener Motion Generation from Speaker Utterance aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance.<n>We present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness.<n>We propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses.
  arXiv  Detail & Related papers  (2026-03-16T10:37:42Z)
• Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning [66.52010873968383]
  We introduce a conversational agent that interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training.<n>The experimental results across four widely used conversational benchmarks demonstrate the effectiveness of our methods.
  arXiv  Detail & Related papers  (2026-01-19T14:55:54Z)
• Reinforcement Learning from User Feedback [28.335218244885706]
  We introduce Reinforcement Learning from User Feedback (RLUF), a framework for aligning large language models with user preferences.<n>We train a reward model, P[Love], to predict the likelihood that an LLM response will receive a Love Reaction.<n>We show that P[Love] is predictive of increased positive feedback and serves as a reliable offline evaluator of future user behavior.
  arXiv  Detail & Related papers  (2025-05-20T22:14:44Z)
• Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated User [117.82681846559909]
  Conversational recommendation systems (CRSs) use multi-turn interaction to capture user preferences and provide personalized recommendations.<n>We propose a generative reward model based simulated user, named GRSU, for automatic interaction with CRSs.
  arXiv  Detail & Related papers  (2025-04-29T06:37:30Z)
• Exploring the Impact of Personality Traits on Conversational Recommender Systems: A Simulation with Large Language Models [70.180385882195]
  This paper introduces a personality-aware user simulation for Conversational Recommender Systems (CRSs)<n>The user agent induces customizable personality traits and preferences, while the system agent possesses the persuasion capability to simulate realistic interaction in CRSs.<n> Experimental results demonstrate that state-of-the-art LLMs can effectively generate diverse user responses aligned with specified personality traits.
  arXiv  Detail & Related papers  (2025-04-09T13:21:17Z)
• Stop Playing the Guessing Game! Target-free User Simulation for Evaluating Conversational Recommender Systems [21.275452863162936]
  PEPPER is an evaluation protocol with target-free user simulators constructed from real-user interaction histories and reviews.<n> PEPPER enables realistic user-CRS dialogues without falling into simplistic guessing games.<n> PEPPER presents detailed measures for comprehensively evaluating the preference elicitation capabilities of CRSs.
  arXiv  Detail & Related papers  (2024-11-25T07:36:20Z)
• Towards Empathetic Conversational Recommender Systems [77.53167131692]
  We propose an empathetic conversational recommender (ECR) framework. ECR contains two main modules: emotion-aware item recommendation and emotion-aligned response generation. Our experiments on the ReDial dataset validate the efficacy of our framework in enhancing recommendation accuracy and improving user satisfaction.
  arXiv  Detail & Related papers  (2024-08-30T15:43:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.