SAPIEN: Affective Virtual Agents Powered by Large Language Models

• URL: http://arxiv.org/abs/2308.03022v1
• Date: Sun, 6 Aug 2023 05:13:16 GMT
• Title: SAPIEN: Affective Virtual Agents Powered by Large Language Models
• Authors: Masum Hasan, Cengiz Ozel, Sammy Potter and Ehsan Hoque
• Abstract summary: We introduce SAPIEN, a platform for high-fidelity virtual agents driven by large language models. The platform allows users to customize their virtual agent's personality, background, and conversation premise. After the virtual meeting, the user can choose to get the conversation analyzed and receive actionable feedback on their communication skills.
• Score: 2.423280064224919
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this demo paper, we introduce SAPIEN, a platform for high-fidelity virtual agents driven by large language models that can hold open domain conversations with users in 13 different languages, and display emotions through facial expressions and voice. The platform allows users to customize their virtual agent's personality, background, and conversation premise, thus providing a rich, immersive interaction experience. Furthermore, after the virtual meeting, the user can choose to get the conversation analyzed and receive actionable feedback on their communication skills. This paper illustrates an overview of the platform and discusses the various application domains of this technology, ranging from entertainment to mental health, communication training, language learning, education, healthcare, and beyond. Additionally, we consider the ethical implications of such realistic virtual agent representations and the potential challenges in ensuring responsible use.

Related papers

• RITA: A Real-time Interactive Talking Avatars Framework [6.060251768347276]
  RITA presents a high-quality real-time interactive framework built upon generative models. Our framework enables the transformation of user-uploaded photos into digital avatars that can engage in real-time dialogue interactions.
  arXiv  Detail & Related papers  (2024-06-18T22:53:15Z)
• VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications [2.5022287664959446]
  This study introduces a pioneering approach utilizing Visual Language Models within VR environments to enhance user interaction and task efficiency. Our system facilitates real-time, intuitive user interactions through natural language processing, without relying on visual text instructions.
  arXiv  Detail & Related papers  (2024-05-19T12:56:00Z)
• Agent AI: Surveying the Horizons of Multimodal Interaction [83.18367129924997]
  "Agent AI" is a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data. We envision a future where people can easily create any virtual reality or simulated scene and interact with agents embodied within the virtual environment.
  arXiv  Detail & Related papers  (2024-01-07T19:11:18Z)
• Prompt-to-OS (P2OS): Revolutionizing Operating Systems and Human-Computer Interaction with Integrated AI Generative Models [10.892991111926573]
  We present a paradigm for human-computer interaction that revolutionizes the traditional notion of an operating system. Within this innovative framework, user requests issued to the machine are handled by an interconnected ecosystem of generative AI models. This visionary concept raises significant challenges, including privacy, security, trustability, and the ethical use of generative models.
  arXiv  Detail & Related papers  (2023-10-07T17:16:34Z)
• Let's Give a Voice to Conversational Agents in Virtual Reality [2.7470819871568506]
  We present an open-source architecture with the goal of simplifying the development of conversational agents in virtual environments. We present two conversational prototypes operating in the digital health domain developed in Unity for both non-immersive displays and VR headsets.
  arXiv  Detail & Related papers  (2023-08-04T18:51:38Z)
• FaceChat: An Emotion-Aware Face-to-face Dialogue Framework [58.67608580694849]
  FaceChat is a web-based dialogue framework that enables emotionally-sensitive and face-to-face conversations. System has a wide range of potential applications, including counseling, emotional support, and personalized customer service.
  arXiv  Detail & Related papers  (2023-03-08T20:45:37Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction [50.986371459817256]
  We propose a novel textitVirtual InteRacTion mechanism, termed as VIRT, to enable full and deep interaction modeling in representation-based models. VIRT asks representation-based encoders to conduct virtual interactions to mimic the behaviors as interaction-based models do.
  arXiv  Detail & Related papers  (2021-12-08T09:49:28Z)
• Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents [60.27066549589362]
  Social language used by human agents is associated with greater users' responsiveness and task completion. The model uses a sequence-to-sequence deep learning architecture, extended with a social language understanding element. Evaluation in terms of content preservation and social language level using both human judgment and automatic linguistic measures shows that the model can generate responses that enable agents to address users' issues in a more socially appropriate way.
  arXiv  Detail & Related papers  (2020-12-29T08:22:48Z)
• SPA: Verbal Interactions between Agents and Avatars in Shared Virtual Environments using Propositional Planning [61.335252950832256]
  Sense-Plan-Ask, or SPA, generates plausible verbal interactions between virtual human-like agents and user avatars in shared virtual environments. We find that our algorithm creates a small runtime cost and enables agents to complete their goals more effectively than agents without the ability to leverage natural-language communication.
  arXiv  Detail & Related papers  (2020-02-08T23:15:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.