Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

• URL: http://arxiv.org/abs/2406.12707v1
• Date: Tue, 18 Jun 2024 15:19:51 GMT
• Title: Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction
• Authors: Haoqiu Yan, Yongxin Zhu, Kai Zheng, Bing Liu, Haoyu Cao, Deqiang Jiang, Linli Xu,
• Abstract summary: PerceptiveAgent is an empathetic multi-modal dialogue system designed to discern deeper or more subtle meanings. PerceptiveAgent perceives acoustic information from input speech and generates empathetic responses based on speaking styles described in natural language.
• Score: 23.115506530649988
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Model (LLM)-enhanced agents become increasingly prevalent in Human-AI communication, offering vast potential from entertainment to professional domains. However, current multi-modal dialogue systems overlook the acoustic information present in speech, which is crucial for understanding human communication nuances. This oversight can lead to misinterpretations of speakers' intentions, resulting in inconsistent or even contradictory responses within dialogues. To bridge this gap, in this paper, we propose PerceptiveAgent, an empathetic multi-modal dialogue system designed to discern deeper or more subtle meanings beyond the literal interpretations of words through the integration of speech modality perception. Employing LLMs as a cognitive core, PerceptiveAgent perceives acoustic information from input speech and generates empathetic responses based on speaking styles described in natural language. Experimental results indicate that PerceptiveAgent excels in contextual understanding by accurately discerning the speakers' true intentions in scenarios where the linguistic meaning is either contrary to or inconsistent with the speaker's true feelings, producing more nuanced and expressive spoken dialogues. Code is publicly available at: \url{https://github.com/Haoqiu-Yan/PerceptiveAgent}.

Related papers

• Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations [58.65755268815283]
  Many real dialogues are interactive, meaning an agent's utterances will influence their conversational partner, elicit information, or change their opinion. We use this fact to rewrite and augment existing suboptimal data, and train via offline reinforcement learning (RL) an agent that outperforms both prompting and learning from unaltered human demonstrations. Our results in a user study with real humans show that our approach greatly outperforms existing state-of-the-art dialogue agents.
  arXiv  Detail & Related papers  (2024-11-07T21:37:51Z)
• Improving Speaker Diarization using Semantic Information: Joint Pairwise Constraints Propagation [53.01238689626378]
  We propose a novel approach to leverage semantic information in speaker diarization systems. We introduce spoken language understanding modules to extract speaker-related semantic information. We present a novel framework to integrate these constraints into the speaker diarization pipeline.
  arXiv  Detail & Related papers  (2023-09-19T09:13:30Z)
• Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents [5.244401764969407]
  Embodied agents, in the form of virtual agents or social robots, are rapidly becoming more widespread. We propose a novel framework that can generate sequences of joint angles from the speech text and speech audio utterances.
  arXiv  Detail & Related papers  (2023-09-17T18:46:25Z)
• Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind [4.052000839878213]
  We model a visually grounded referential game between a knowledgeable speaker and a listener with more limited visual and linguistic experience. We endow our speaker with the ability to adapt its referring expressions via a simulation module that monitors the effectiveness of planned utterances from the listener's perspective.
  arXiv  Detail & Related papers  (2023-05-31T15:17:28Z)
• Context-Dependent Embedding Utterance Representations for Emotion Recognition in Conversations [1.8126187844654875]
  We approach Emotion Recognition in Conversations leveraging the conversational context. We propose context-dependent embedding representations of each utterance. The effectiveness of our approach is validated on the open-domain DailyDialog dataset and on the task-oriented EmoWOZ dataset.
  arXiv  Detail & Related papers  (2023-04-17T12:37:57Z)
• deep learning of segment-level feature representation for speech emotion recognition in conversations [9.432208348863336]
  We propose a conversational speech emotion recognition method to deal with capturing attentive contextual dependency and speaker-sensitive interactions. First, we use a pretrained VGGish model to extract segment-based audio representation in individual utterances. Second, an attentive bi-directional recurrent unit (GRU) models contextual-sensitive information and explores intra- and inter-speaker dependencies jointly.
  arXiv  Detail & Related papers  (2023-02-05T16:15:46Z)
• Channel-aware Decoupling Network for Multi-turn Dialogue Comprehension [81.47133615169203]
  We propose compositional learning for holistic interaction across utterances beyond the sequential contextualization from PrLMs. We employ domain-adaptive training strategies to help the model adapt to the dialogue domains. Experimental results show that our method substantially boosts the strong PrLM baselines in four public benchmark datasets.
  arXiv  Detail & Related papers  (2023-01-10T13:18:25Z)
• Filling the Gap of Utterance-aware and Speaker-aware Representation for Multi-turn Dialogue [76.88174667929665]
  A multi-turn dialogue is composed of multiple utterances from two or more different speaker roles. In the existing retrieval-based multi-turn dialogue modeling, the pre-trained language models (PrLMs) as encoder represent the dialogues coarsely. We propose a novel model to fill such a gap by modeling the effective utterance-aware and speaker-aware representations entailed in a dialogue history.
  arXiv  Detail & Related papers  (2020-09-14T15:07:19Z)
• Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness [62.55060760615656]
  Recent models tackling consistency often train with additional Natural Language Inference (NLI) labels or attach trained extra modules to the generative agent for maintaining consistency. Inspired by social cognition and pragmatics, we endow existing dialogue agents with public self-consciousness on the fly through an imaginary listener. Our approach, based on the Rational Speech Acts framework, can enforce dialogue agents to refrain from uttering contradiction.
  arXiv  Detail & Related papers  (2020-04-13T08:16:16Z)
• You Impress Me: Dialogue Generation via Mutual Persona Perception [62.89449096369027]
  The research in cognitive science suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose P2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding.
  arXiv  Detail & Related papers  (2020-04-11T12:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.