Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera

• URL: http://arxiv.org/abs/2408.07982v1
• Date: Thu, 15 Aug 2024 07:03:00 GMT
• Title: Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera
• Authors: Hiroki Tanioka, Tetsushi Ueta, Masahiko Sano,
• Abstract summary: Methods for AI agents to recognize emotions from the user's facial expressions have not been studied. We examined whether or not LLM-based AI agents can interact with users according to their emotional states by capturing the user in dialogue with a camera. Results confirmed that AI agents can have conversations according to the emotional state for emotional states with relatively high scores, such as Happy and Angry.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The performance of ChatGPT\copyright{} and other LLMs has improved tremendously, and in online environments, they are increasingly likely to be used in a wide variety of situations, such as ChatBot on web pages, call center operations using voice interaction, and dialogue functions using agents. In the offline environment, multimodal dialogue functions are also being realized, such as guidance by Artificial Intelligence agents (AI agents) using tablet terminals and dialogue systems in the form of LLMs mounted on robots. In this multimodal dialogue, mutual emotion recognition between the AI and the user will become important. So far, there have been methods for expressing emotions on the part of the AI agent or for recognizing them using textual or voice information of the user's utterances, but methods for AI agents to recognize emotions from the user's facial expressions have not been studied. In this study, we examined whether or not LLM-based AI agents can interact with users according to their emotional states by capturing the user in dialogue with a camera, recognizing emotions from facial expressions, and adding such emotion information to prompts. The results confirmed that AI agents can have conversations according to the emotional state for emotional states with relatively high scores, such as Happy and Angry.

Related papers

• Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations [58.65755268815283]
  Many real dialogues are interactive, meaning an agent's utterances will influence their conversational partner, elicit information, or change their opinion. We use this fact to rewrite and augment existing suboptimal data, and train via offline reinforcement learning (RL) an agent that outperforms both prompting and learning from unaltered human demonstrations. Our results in a user study with real humans show that our approach greatly outperforms existing state-of-the-art dialogue agents.
  arXiv  Detail & Related papers  (2024-11-07T21:37:51Z)
• LLM Roleplay: Simulating Human-Chatbot Interaction [52.03241266241294]
  We propose a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction. Our method can simulate human-chatbot dialogues with a high indistinguishability rate.
  arXiv  Detail & Related papers  (2024-07-04T14:49:46Z)
• Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction [23.115506530649988]
  PerceptiveAgent is an empathetic multi-modal dialogue system designed to discern deeper or more subtle meanings. PerceptiveAgent perceives acoustic information from input speech and generates empathetic responses based on speaking styles described in natural language.
  arXiv  Detail & Related papers  (2024-06-18T15:19:51Z)
• Think out Loud: Emotion Deducing Explanation in Dialogues [57.90554323226896]
  We propose a new task "Emotion Deducing Explanation in Dialogues" (EDEN) EDEN recognizes emotion and causes in an explicitly thinking way. It can help Large Language Models (LLMs) achieve better recognition of emotions and causes.
  arXiv  Detail & Related papers  (2024-06-07T08:58:29Z)
• Research on emotionally intelligent dialogue generation based on automatic dialogue system [10.064417058641979]
  This study integrates emotional intelligence technology into automated dialogue systems. It creates a dialogue generation model with emotional intelligence through deep learning and natural language processing techniques.
  arXiv  Detail & Related papers  (2024-04-17T14:55:03Z)
• FaceChat: An Emotion-Aware Face-to-face Dialogue Framework [58.67608580694849]
  FaceChat is a web-based dialogue framework that enables emotionally-sensitive and face-to-face conversations. System has a wide range of potential applications, including counseling, emotional support, and personalized customer service.
  arXiv  Detail & Related papers  (2023-03-08T20:45:37Z)
• A Unified Framework for Emotion Identification and Generation in Dialogues [5.102770724328495]
  We propose a multi-task framework that jointly identifies the emotion of a given dialogue and generates response in accordance to the identified emotion. We employ a BERT based network for creating an empathetic system and use a mixed objective function that trains the end-to-end network with both the classification and generation loss.
  arXiv  Detail & Related papers  (2022-05-31T02:58:49Z)
• CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AI [48.67259855309959]
  Most existing datasets for conversational AI ignore human personalities and emotions. We propose CPED, a large-scale Chinese personalized and emotional dialogue dataset. CPED contains more than 12K dialogues of 392 speakers from 40 TV shows.
  arXiv  Detail & Related papers  (2022-05-29T17:45:12Z)
• Towards Multi-Turn Empathetic Dialogs with Positive Emotion Elicitation [39.747587984500406]
  This paper presents a novel task of empathetic dialog generation with positive emotion elicitation. The agent conducts empathetic responses along with the target of eliciting the user's positive emotions in the multi-turn dialog. We collect a large-scale emotional dialog dataset with positive emotion elicitation, called PosEmoDial.
  arXiv  Detail & Related papers  (2022-04-22T05:32:08Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion in Task-Oriented Dialogue Systems [3.3010169113961325]
  EmoWOZ is a large-scale manually emotion-annotated corpus of task-oriented dialogues. It contains more than 11K dialogues with more than 83K emotion annotations of user utterances. We propose a novel emotion labelling scheme, which is tailored to task-oriented dialogues.
  arXiv  Detail & Related papers  (2021-09-10T15:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.