Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning

• URL: http://arxiv.org/abs/2606.00851v1
• Date: Sat, 30 May 2026 18:53:35 GMT
• Title: Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning
• Abstract summary: Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately.<n>We introduce Sympatheia, a speech-to-speech dialogue framework conditioned on affect inferred from the user's speech.
• Score: 17.756766677106388
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately, yet everyday speech often carries weak, neutral, or ambiguous affective cues. To address this, we introduce Sympatheia, a speech-to-speech dialogue framework conditioned on affect inferred from the user's speech and, when available, explicit affect specifications provided as a continuous valence--arousal (VA) control signal by a multimodal sensing module or user interface. To train our model, we construct Sympatheia-18k, an emotion-conditioned synthetic spoken dialogue corpus with 12 emotion anchors. This dataset includes an emotional split for learning affective speech behavior, and a neutral split that pairs emotionally neutral queries with multiple emotion-conditioned responses to isolate explicit emotion control in emotionally ambiguous cases. Empirical results show that Sympatheia outperforms speech conversational baselines in generating responses whose semantic content and spoken delivery are both emotionally appropriate. We further show that the same VA interface can integrate emotion estimates from diverse sensing modules, including facial expression, biosignals, and textual affect descriptions, improving response alignment when speech alone provides limited emotional evidence. These results suggest that continuous affect conditioning is an effective practical step for building emotionally adaptive voice assistants.

Related papers

• On the Emotion Understanding of Synthesized Speech [63.13411068766772]
  Emotion is a core paralinguistic feature in voice interaction.<n>Current Speech Emotion Recognition (SER) models can not generalize to synthesized speech.<n>generative Speech Language Models (SLMs) tend to infer emotion from textual semantics while ignoring paralinguistic cues.
  arXiv  Detail & Related papers  (2026-03-17T13:11:14Z)
• A Unified Spoken Language Model with Injected Emotional-Attribution Thinking for Human-like Interaction [50.05919688888947]
  This paper presents a unified spoken language model for emotional intelligence, enhanced by a novel data construction strategy termed Injected Emotional-Attribution Thinking (IEAT)<n>IEAT incorporates user emotional states and their underlying causes into the model's internal reasoning process, enabling emotion-aware reasoning to be internalized rather than treated as explicit supervision.<n> Experiments on the Human-like Spoken Dialogue Systems Challenge (HumDial) Emotional Intelligence benchmark demonstrate that the proposed approach achieves top-ranked performance across emotional trajectory modeling, emotional reasoning, and empathetic response generation.
  arXiv  Detail & Related papers  (2026-01-08T14:07:30Z)
• Semantic Differentiation in Speech Emotion Recognition: Insights from Descriptive and Expressive Speech Roles [4.516156697420418]
  Speech Emotion Recognition (SER) is essential for improving human-computer interaction.<n>We distinguish between descriptive semantics, which represents the contextual content of speech, and expressive semantics, which reflects the speaker's emotional state.<n>Our findings inform SER applications in human-AI interaction and pave the way for more context-aware AI systems.
  arXiv  Detail & Related papers  (2025-10-03T14:42:35Z)
• EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems [25.920610612373608]
  EMO-Reasoning is a benchmark for assessing emotional coherence in dialogue systems.<n>We propose the Cross-turn Emotion Reasoning Score to assess the emotion transitions in multi-turn dialogues.
  arXiv  Detail & Related papers  (2025-08-25T03:01:40Z)
• DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech [49.128847336227636]
  Cross-speaker emotion transfer in speech synthesis relies on extracting speaker-independent emotion embeddings for accurate emotion modeling.<n>We propose DiEmo-TTS, a self-supervised distillation method to minimize emotional information loss and preserve speaker identity.
  arXiv  Detail & Related papers  (2025-05-26T08:47:39Z)
• AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues [37.96886343501444]
  We present AV-EmoDialog, a dialogue system designed to exploit verbal and non-verbal information from users' audio-visual inputs to generate more responsive and empathetic interactions.<n> AV-EmoDialog systematically exploits the emotional cues in audio-visual dialogues; extracting speech content and emotional tones from speech, analyzing fine-grained facial expressions from visuals, and integrating these cues to generate emotionally aware responses in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-12-23T05:24:26Z)
• Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion [81.1492897350032]
  Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. We propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion.
  arXiv  Detail & Related papers  (2023-12-29T08:06:45Z)
• AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis [13.918119853846838]
  Affect is an emotional characteristic encompassing valence, arousal, and intensity, and is a crucial attribute for enabling authentic conversations. We propose AffectEcho, an emotion translation model, that uses a Vector Quantized codebook to model emotions within a quantized space. We demonstrate the effectiveness of our approach in controlling the emotions of generated speech while preserving identity, style, and emotional cadence unique to each speaker.
  arXiv  Detail & Related papers  (2023-08-16T06:28:29Z)
• Learning Emotional Representations from Imbalanced Speech Data for Speech Emotion Recognition and Emotional Text-to-Speech [1.4986031916712106]
  Speech emotional representations play a key role in Speech Emotion Recognition (SER) and Emotional Text-To-Speech (TTS) tasks. Models might overfit to the majority Neutral class and fail to produce robust and effective emotional representations. We use augmentation approaches to train the model and enable it to extract effective and generalizable emotional representations from imbalanced datasets.
  arXiv  Detail & Related papers  (2023-06-09T07:04:56Z)
• Emotion Intensity and its Control for Emotional Voice Conversion [77.05097999561298]
  Emotional voice conversion (EVC) seeks to convert the emotional state of an utterance while preserving the linguistic content and speaker identity. In this paper, we aim to explicitly characterize and control the intensity of emotion. We propose to disentangle the speaker style from linguistic content and encode the speaker style into a style embedding in a continuous space that forms the prototype of emotion embedding.
  arXiv  Detail & Related papers  (2022-01-10T02:11:25Z)
• Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability [82.39099867188547]
  Emotional text-to-speech synthesis (ETTS) has seen much progress in recent years. We propose a new interactive training paradigm for ETTS, denoted as i-ETTS. We formulate an iterative training strategy with reinforcement learning to ensure the quality of i-ETTS optimization.
  arXiv  Detail & Related papers  (2021-04-03T13:52:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.