Closing the Affective Loop: Multimodal Speaker-Listener Emotion-Dynamics-Aware Empathetic Social Robots

• URL: http://arxiv.org/abs/2608.16686v1
• Date: Mon, 17 Aug 2026 15:08:38 GMT
• Title: Closing the Affective Loop: Multimodal Speaker-Listener Emotion-Dynamics-Aware Empathetic Social Robots
• Abstract summary: AffectLoop is a speaker-listener emotion-dynamics-aware spoken dialogue system implemented on the Misty II robot.<n>The system generates a short spoken empathetic response together with emotionally congruent embodied behavior.<n>Preliminary results suggest that explicitly modeling both speaker emotional dynamics and listener affective state can improve embodied empathetic interaction.
• Score: 19.640979258341904
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Empathetic social robots should respond not only to what users say, but also to how their emotions dynamically evolve during interaction. However, existing empathetic dialogue systems are often text-centered and primarily model empathy as a one-way mapping from the user's emotion to the system response, limiting their ability to capture embodied speaker--listener affective exchange. We present AffectLoop, a multimodal speaker-listener emotion-dynamics-aware spoken dialogue system implemented on the Misty II robot. The system tracks the speaker's verbal and facial affective dynamics, estimates the robot listener's own verbal and behavioral affective state, and conditions LLM-based response generation on both affective streams. The robot then generates a short spoken empathetic response together with emotionally congruent embodied behavior, forming a closed speaker--listener affective loop. We evaluate the system in a pilot within-subject study with five participants, comparing it with an otherwise identical utterance-conditioned baseline that omits the speaker- and listener-affective-state inputs. The proposed system received higher overall impression ratings, especially for empathetic response and user satisfaction. Post-hoc log analysis further showed higher speaker-listener affective alignment and stronger valence-based distress recovery. These preliminary results suggest that explicitly modeling both speaker emotional dynamics and listener affective state can improve embodied empathetic interaction.

Related papers

• Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning [17.756766677106388]
  Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately.<n>We introduce Sympatheia, a speech-to-speech dialogue framework conditioned on affect inferred from the user's speech.
  arXiv  Detail & Related papers  (2026-05-30T18:53:35Z)
• Evaluating multimodal emotion recognition in proactive conversational agents: A user study [0.0]
  This article presents a multimodal emotion recognition module integrated into a proactive Socially Interactive Agent (SIA) powered by generative artificial intelligence.<n>The system evaluates real-time affective states through two distinct channels: a computer vision-based facial recognition module and a semantic linguistic analysis engine.
  arXiv  Detail & Related papers  (2026-04-06T15:11:43Z)
• Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialogue [53.95386201009769]
  We introduce EmpathyEval, a descriptive natural-language-based evaluation model for assessing empathetic quality in spoken dialogues.<n>We propose ReEmpathy, an end-to-end Spoken Language Models that enhances empathetic dialogue through a novel Empathetic Self-Reflective Alternating Inference mechanism.
  arXiv  Detail & Related papers  (2026-01-26T09:04:50Z)
• A Unified Spoken Language Model with Injected Emotional-Attribution Thinking for Human-like Interaction [50.05919688888947]
  This paper presents a unified spoken language model for emotional intelligence, enhanced by a novel data construction strategy termed Injected Emotional-Attribution Thinking (IEAT)<n>IEAT incorporates user emotional states and their underlying causes into the model's internal reasoning process, enabling emotion-aware reasoning to be internalized rather than treated as explicit supervision.<n> Experiments on the Human-like Spoken Dialogue Systems Challenge (HumDial) Emotional Intelligence benchmark demonstrate that the proposed approach achieves top-ranked performance across emotional trajectory modeling, emotional reasoning, and empathetic response generation.
  arXiv  Detail & Related papers  (2026-01-08T14:07:30Z)
• EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems [25.920610612373608]
  EMO-Reasoning is a benchmark for assessing emotional coherence in dialogue systems.<n>We propose the Cross-turn Emotion Reasoning Score to assess the emotion transitions in multi-turn dialogues.
  arXiv  Detail & Related papers  (2025-08-25T03:01:40Z)
• Personality-affected Emotion Generation in Dialog Systems [67.40609683389947]
  We propose a new task, Personality-affected Emotion Generation, to generate emotion based on the personality given to the dialog system. We analyze the challenges in this task, i.e., (1) heterogeneously integrating personality and emotional factors and (2) extracting multi-granularity emotional information in the dialog context. Results suggest that by adopting our method, the emotion generation performance is improved by 13% in macro-F1 and 5% in weighted-F1 from the BERT-base model.
  arXiv  Detail & Related papers  (2024-04-03T08:48:50Z)
• Emotional Listener Portrait: Realistic Listener Motion Simulation in Conversation [50.35367785674921]
  Listener head generation centers on generating non-verbal behaviors of a listener in reference to the information delivered by a speaker. A significant challenge when generating such responses is the non-deterministic nature of fine-grained facial expressions during a conversation. We propose the Emotional Listener Portrait (ELP), which treats each fine-grained facial motion as a composition of several discrete motion-codewords. Our ELP model can not only automatically generate natural and diverse responses toward a given speaker via sampling from the learned distribution but also generate controllable responses with a predetermined attitude.
  arXiv  Detail & Related papers  (2023-09-29T18:18:32Z)
• Affective Decoding for Empathetic Response Generation [8.391383696266704]
  We propose a technique called Affective Decoding for empathetic response generation. Our method can effectively incorporate emotion signals during each decoding step. Our models are perceived to be more empathetic by human evaluations.
  arXiv  Detail & Related papers  (2021-08-18T11:48:40Z)
• Emotion-aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction [55.47134146639492]
  This article proposes a unifed end-to-end neural architecture, which is capable of simultaneously encoding the semantics and the emotions in a post. Experiments on real-world data demonstrate that the proposed method outperforms the state-of-the-art methods in terms of both content coherence and emotion appropriateness.
  arXiv  Detail & Related papers  (2021-06-06T06:26:15Z)
• Disambiguating Affective Stimulus Associations for Robot Perception and Dialogue [67.89143112645556]
  We provide a NICO robot with the ability to learn the associations between a perceived auditory stimulus and an emotional expression. NICO is able to do this for both individual subjects and specific stimuli, with the aid of an emotion-driven dialogue system. The robot is then able to use this information to determine a subject's enjoyment of perceived auditory stimuli in a real HRI scenario.
  arXiv  Detail & Related papers  (2021-03-05T20:55:48Z)
• Generating Emotionally Aligned Responses in Dialogues using Affect Control Theory [15.848210524718219]
  Affect Control Theory (ACT) is a socio-mathematical model of emotions for human-human interactions. We investigate how ACT can be used to develop affect-aware neural conversational agents.
  arXiv  Detail & Related papers  (2020-03-07T19:31:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.