The Importance of Multimodal Emotion Conditioning and Affect Consistency for Embodied Conversational Agents

• URL: http://arxiv.org/abs/2309.15311v2
• Date: Wed, 6 Dec 2023 21:56:27 GMT
• Title: The Importance of Multimodal Emotion Conditioning and Affect Consistency for Embodied Conversational Agents
• Authors: Che-Jui Chang, Samuel S. Sohn, Sen Zhang, Rajath Jayashankar, Muhammad Usman, Mubbasir Kapadia
• Abstract summary: We propose a conceptual framework that aims to increase the perception of affects by generating multimodal behaviors conditioned on a consistent driving affect. Our statistical analysis suggests that making a modality affect-inconsistent significantly decreases the perception of driving affects.
• Score: 12.102955731466457
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Previous studies regarding the perception of emotions for embodied virtual agents have shown the effectiveness of using virtual characters in conveying emotions through interactions with humans. However, creating an autonomous embodied conversational agent with expressive behaviors presents two major challenges. The first challenge is the difficulty of synthesizing the conversational behaviors for each modality that are as expressive as real human behaviors. The second challenge is that the affects are modeled independently, which makes it difficult to generate multimodal responses with consistent emotions across all modalities. In this work, we propose a conceptual framework, ACTOR (Affect-Consistent mulTimodal behaviOR generation), that aims to increase the perception of affects by generating multimodal behaviors conditioned on a consistent driving affect. We have conducted a user study with 199 participants to assess how the average person judges the affects perceived from multimodal behaviors that are consistent and inconsistent with respect to a driving affect. The result shows that among all model conditions, our affect-consistent framework receives the highest Likert scores for the perception of driving affects. Our statistical analysis suggests that making a modality affect-inconsistent significantly decreases the perception of driving affects. We also observe that multimodal behaviors conditioned on consistent affects are more expressive compared to behaviors with inconsistent affects. Therefore, we conclude that multimodal emotion conditioning and affect consistency are vital to enhancing the perception of affects for embodied conversational agents.

Related papers

• Interact2Ar: Full-Body Human-Human Interaction Generation via Autoregressive Diffusion Models [80.28579390566298]
  We introduce Interact2Ar, a text-conditioned autoregressive diffusion model for generating full-body, human-human interactions.<n>Hand kinematics are incorporated through dedicated parallel branches, enabling high-fidelity full-body generation.<n>Our model enables a series of downstream applications, including temporal motion composition, real-time adaptation to disturbances, and extension beyond dyadic to multi-person scenarios.
  arXiv  Detail & Related papers  (2025-12-22T18:59:50Z)
• Human Cognitive Biases in Explanation-Based Interaction: The Case of Within and Between Session Order Effect [46.80756527630539]
  Explanatory Interactive Learning (XIL) is a powerful interactive learning framework designed to enable users to customize and correct AI models by interacting with their explanations.<n>Recent studies have raised concerns that explanatory interaction may trigger order effects, a well-known cognitive bias in which the sequence of presented items influences users' trust and, critically, the quality of their feedback.<n>To clarify the interplay between order effects and explanatory interaction, we ran two larger-scale user studies designed to mimic common XIL tasks.
  arXiv  Detail & Related papers  (2025-12-04T12:59:54Z)
• DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios [57.327907850766785]
  characterization of deception across realistic real-world scenarios remains underexplored.<n>We establish DeceptionBench, the first benchmark that systematically evaluates how deceptive tendencies manifest across different domains.<n>On the intrinsic dimension, we explore whether models exhibit self-interested egoistic tendencies or sycophantic behaviors that prioritize user appeasement.<n>We incorporate sustained multi-turn interaction loops to construct a more realistic simulation of real-world feedback dynamics.
  arXiv  Detail & Related papers  (2025-10-17T10:14:26Z)
• Modelling the Interplay of Eye-Tracking Temporal Dynamics and Personality for Emotion Detection in Face-to-Face Settings [1.2600839346487007]
  This work presents a personality-aware multimodal framework that integrates eye-tracking sequences, Big Five personality traits, and contextual stimulus cues to predict both perceived and felt emotions.<n>Results show that stimulus cues strongly enhance perceived-emotion predictions, while personality traits provide the largest improvements for felt emotion recognition.
  arXiv  Detail & Related papers  (2025-09-19T16:05:23Z)
• The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs [60.15472325639723]
  Personality traits have long been studied as predictors of human behavior.<n>Recent advances in Large Language Models (LLMs) suggest similar patterns may emerge in artificial systems.
  arXiv  Detail & Related papers  (2025-09-03T21:27:10Z)
• CauESC: A Causal Aware Model for Emotional Support Conversation [79.4451588204647]
  Existing approaches ignore the emotion causes of the distress. They focus on the seeker's own mental state rather than the emotional dynamics during interaction between speakers. We propose a novel framework CauESC, which firstly recognizes the emotion causes of the distress, as well as the emotion effects triggered by the causes.
  arXiv  Detail & Related papers  (2024-01-31T11:30:24Z)
• Decoding Susceptibility: Modeling Misbelief to Misinformation Through a Computational Approach [61.04606493712002]
  Susceptibility to misinformation describes the degree of belief in unverifiable claims that is not observable. Existing susceptibility studies heavily rely on self-reported beliefs. We propose a computational approach to model users' latent susceptibility levels.
  arXiv  Detail & Related papers  (2023-11-16T07:22:56Z)
• Dynamic Causal Disentanglement Model for Dialogue Emotion Detection [77.96255121683011]
  We propose a Dynamic Causal Disentanglement Model based on hidden variable separation. This model effectively decomposes the content of dialogues and investigates the temporal accumulation of emotions. Specifically, we propose a dynamic temporal disentanglement model to infer the propagation of utterances and hidden variables.
  arXiv  Detail & Related papers  (2023-09-13T12:58:09Z)
• HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer [38.92436852096451]
  Cross-person memory Transformer (CPM-T) framework is able to explicitly model affective dynamics. CPM-T framework maintains memory modules to store and update the contexts within the conversation window. We evaluate the effectiveness and generalizability of our approach on three publicly available datasets for joint engagement, rapport, and human beliefs prediction tasks.
  arXiv  Detail & Related papers  (2023-05-21T06:43:35Z)
• Expanding the Role of Affective Phenomena in Multimodal Interaction Research [57.069159905961214]
  We examined over 16,000 papers from selected conferences in multimodal interaction, affective computing, and natural language processing. We identify 910 affect-related papers and present our analysis of the role of affective phenomena in these papers. We find limited research on how affect and emotion predictions might be used by AI systems to enhance machine understanding of human social behaviors and cognitive states.
  arXiv  Detail & Related papers  (2023-05-18T09:08:39Z)
• Computational Empathy Counteracts the Negative Effects of Anger on Creative Problem Solving [2.322052136673525]
  We introduce a computational empathy intervention based on context-specific affective mimicry and perspective taking by a virtual agent appearing in the form of a well-dressed polar bear. We examine how anger and empathy influence participants' performance in solving a word game based on Wordle.
  arXiv  Detail & Related papers  (2022-08-15T13:31:49Z)
• Analysing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study [6.4985954299863]
  This study is concerned with the analysis of the direction of emotional influence in dyadic dialogue based on facial expressions only. We exploit computer vision capabilities along with causal inference theory for quantitative verification of hypotheses on the direction of emotional influence.
  arXiv  Detail & Related papers  (2020-12-16T07:52:35Z)
• Intrinsic motivation in virtual assistant interaction for fostering spontaneous interactions [3.420509295457138]
  This study aims to cover intrinsic motivation by taking an affective-engineering approach. A novel motivation model is proposed, in which intrinsic motivation is affected by two factors: expectation of capability and uncertainty. Results of the first experiment showed that high expectation engenders more intrinsically motivated interaction compared with low expectation.
  arXiv  Detail & Related papers  (2020-10-13T14:23:57Z)
• Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition [55.44502358463217]
  We propose a modality-transferable model with emotion embeddings to tackle the aforementioned issues. Our model achieves state-of-the-art performance on most of the emotion categories. Our model also outperforms existing baselines in the zero-shot and few-shot scenarios for unseen emotions.
  arXiv  Detail & Related papers  (2020-09-21T06:10:39Z)
• Towards Persona-Based Empathetic Conversational Models [58.65492299237112]
  Empathetic conversational models have been shown to improve user satisfaction and task outcomes in numerous domains. In Psychology, persona has been shown to be highly correlated to personality, which in turn influences empathy. We propose a new task towards persona-based empathetic conversations and present the first empirical study on the impact of persona on empathetic responding.
  arXiv  Detail & Related papers  (2020-04-26T08:51:01Z)
• Examining the Effects of Emotional Valence and Arousal on Takeover Performance in Conditionally Automated Driving [14.987259704464119]
  In conditionally automated driving, drivers have difficulty in takeover transitions as they become increasingly decoupled from the operational level of driving. This study examined the effects of emotional valence and arousal on drivers takeover timeliness and quality in conditionally automated driving.
  arXiv  Detail & Related papers  (2020-01-13T19:28:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.