Reading the Mood Behind Words: Integrating Prosody-Derived Emotional Context into Socially Responsive VR Agents

• URL: http://arxiv.org/abs/2603.09324v1
• Date: Tue, 10 Mar 2026 07:58:06 GMT
• Title: Reading the Mood Behind Words: Integrating Prosody-Derived Emotional Context into Socially Responsive VR Agents
• Abstract summary: We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context.<n>A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context.
• Score: 13.32446993826821
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In VR interactions with embodied conversational agents, users' emotional intent is often conveyed more by how something is said than by what is said. However, most VR agent pipelines rely on speech-to-text processing, discarding prosodic cues and often producing emotionally incongruent responses despite correct semantics. We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context in an LLM-based conversational agent. A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context to shape response tone and style. Results from a within-subjects VR study (N=30) show significant improvements in dialogue quality, naturalness, engagement, rapport, and human-likeness, with 93.3% of participants preferring the emotion-aware agent.

Related papers

• Semantic Differentiation in Speech Emotion Recognition: Insights from Descriptive and Expressive Speech Roles [4.516156697420418]
  Speech Emotion Recognition (SER) is essential for improving human-computer interaction.<n>We distinguish between descriptive semantics, which represents the contextual content of speech, and expressive semantics, which reflects the speaker's emotional state.<n>Our findings inform SER applications in human-AI interaction and pave the way for more context-aware AI systems.
  arXiv  Detail & Related papers  (2025-10-03T14:42:35Z)
• EmoCAST: Emotional Talking Portrait via Emotive Text Description [56.42674612728354]
  EmoCAST is a diffusion-based framework for precise text-driven emotional synthesis.<n>In appearance modeling, emotional prompts are integrated through a text-guided decoupled emotive module.<n>EmoCAST achieves state-of-the-art performance in generating realistic, emotionally expressive, and audio-synchronized talking-head videos.
  arXiv  Detail & Related papers  (2025-08-28T10:02:06Z)
• AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues [37.96886343501444]
  We present AV-EmoDialog, a dialogue system designed to exploit verbal and non-verbal information from users' audio-visual inputs to generate more responsive and empathetic interactions.<n> AV-EmoDialog systematically exploits the emotional cues in audio-visual dialogues; extracting speech content and emotional tones from speech, analyzing fine-grained facial expressions from visuals, and integrating these cues to generate emotionally aware responses in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-12-23T05:24:26Z)
• Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera [0.0]
  Methods for AI agents to recognize emotions from the user's facial expressions have not been studied.<n>We examined whether or not LLM-based AI agents can interact with users according to their emotional states by capturing the user in dialogue with a camera.<n>Results confirmed that AI agents can have conversations according to the emotional state for emotional states with relatively high scores, such as Happy and Angry.
  arXiv  Detail & Related papers  (2024-08-15T07:03:00Z)
• Personality-affected Emotion Generation in Dialog Systems [67.40609683389947]
  We propose a new task, Personality-affected Emotion Generation, to generate emotion based on the personality given to the dialog system. We analyze the challenges in this task, i.e., (1) heterogeneously integrating personality and emotional factors and (2) extracting multi-granularity emotional information in the dialog context. Results suggest that by adopting our method, the emotion generation performance is improved by 13% in macro-F1 and 5% in weighted-F1 from the BERT-base model.
  arXiv  Detail & Related papers  (2024-04-03T08:48:50Z)
• Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion [81.1492897350032]
  Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. We propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion.
  arXiv  Detail & Related papers  (2023-12-29T08:06:45Z)
• Empathetic Dialogue Generation via Sensitive Emotion Recognition and Sensible Knowledge Selection [47.60224978460442]
  We propose a Serial and Emotion-Knowledge interaction (SEEK) method for empathetic dialogue generation. We use a fine-grained encoding strategy which is more sensitive to the emotion dynamics (emotion flow) in the conversations to predict the emotion-intent characteristic of response. Besides, we design a novel framework to model the interaction between knowledge and emotion to generate more sensible response.
  arXiv  Detail & Related papers  (2022-10-21T03:51:18Z)
• Towards Multi-Turn Empathetic Dialogs with Positive Emotion Elicitation [39.747587984500406]
  This paper presents a novel task of empathetic dialog generation with positive emotion elicitation. The agent conducts empathetic responses along with the target of eliciting the user's positive emotions in the multi-turn dialog. We collect a large-scale emotional dialog dataset with positive emotion elicitation, called PosEmoDial.
  arXiv  Detail & Related papers  (2022-04-22T05:32:08Z)
• Emotion-aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction [55.47134146639492]
  This article proposes a unifed end-to-end neural architecture, which is capable of simultaneously encoding the semantics and the emotions in a post. Experiments on real-world data demonstrate that the proposed method outperforms the state-of-the-art methods in terms of both content coherence and emotion appropriateness.
  arXiv  Detail & Related papers  (2021-06-06T06:26:15Z)
• Disambiguating Affective Stimulus Associations for Robot Perception and Dialogue [67.89143112645556]
  We provide a NICO robot with the ability to learn the associations between a perceived auditory stimulus and an emotional expression. NICO is able to do this for both individual subjects and specific stimuli, with the aid of an emotion-driven dialogue system. The robot is then able to use this information to determine a subject's enjoyment of perceived auditory stimuli in a real HRI scenario.
  arXiv  Detail & Related papers  (2021-03-05T20:55:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.