Relationship between auditory and semantic entrainment using Deep Neural Networks (DNN)

• URL: http://arxiv.org/abs/2312.16599v1
• Date: Wed, 27 Dec 2023 14:50:09 GMT
• Title: Relationship between auditory and semantic entrainment using Deep Neural Networks (DNN)
• Authors: Jay Kejriwal, \v{S}tefan Be\v{n}u\v{s}
• Abstract summary: This study utilized state-of-the-art embeddings such as BERT and TRIpLet Loss network (TRILL) vectors to extract features for measuring semantic and auditory similarities of turns within dialogues. We found people's tendency to entrain on semantic features more when compared to auditory features. The findings of this study might assist in implementing the mechanism of entrainment in human-machine interaction (HMI)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The tendency of people to engage in similar, matching, or synchronized behaviour when interacting is known as entrainment. Many studies examined linguistic (syntactic and lexical structures) and paralinguistic (pitch, intensity) entrainment, but less attention was given to finding the relationship between them. In this study, we utilized state-of-the-art DNN embeddings such as BERT and TRIpLet Loss network (TRILL) vectors to extract features for measuring semantic and auditory similarities of turns within dialogues in two comparable spoken corpora of two different languages. We found people's tendency to entrain on semantic features more when compared to auditory features. Additionally, we found that entrainment in semantic and auditory linguistic features are positively correlated. The findings of this study might assist in implementing the mechanism of entrainment in human-machine interaction (HMI).

Related papers

• Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs [70.3132264719438]
  We aim to fill the research gap by examining how neuron activation is shared across tasks and languages. We classify neurons into four distinct categories based on their responses to a specific input across different languages. Our analysis reveals the following insights: (i) the patterns of neuron sharing are significantly affected by the characteristics of tasks and examples; (ii) neuron sharing does not fully correspond with language similarity; (iii) shared neurons play a vital role in generating responses, especially those shared across all languages.
  arXiv  Detail & Related papers  (2024-06-13T16:04:11Z)
• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks [59.38765771221084]
  We present a physiologically inspired speech recognition architecture compatible and scalable with deep learning frameworks. We show end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronising neural activity to improve recognition performance.
  arXiv  Detail & Related papers  (2024-04-22T09:40:07Z)
• Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognition [54.952250732643115]
  We study Acoustic Word Embeddings (AWEs), a fixed-length feature derived from continuous representations, to explore their advantages in specific tasks. AWEs have previously shown utility in capturing acoustic discriminability. Our findings underscore the acoustic context conveyed by AWEs and showcase the highly competitive Speech Emotion Recognition accuracies.
  arXiv  Detail & Related papers  (2024-02-04T21:24:54Z)
• Unsupervised Auditory and Semantic Entrainment Models with Deep Neural Networks [0.3222802562733786]
  We present an unsupervised deep learning framework that derives meaningful representation from textual features for developing semantic entrainment. The results show that semantic entrainment can be assessed with our model, that models can distinguish between HH and HM interactions and that the two units of analysis for extracting acoustic features provide comparable findings.
  arXiv  Detail & Related papers  (2023-12-22T22:33:54Z)
• Decoding Emotional Experiences in Dyadic Conversations of Married Couples: Leveraging Semantic Similarity through Sentence Embedding [41.94295877935867]
  The present study analyzes verbal conversations of 50 married couples who engage in naturalistic 10-minute conflict and 10-minute positive conversations. Transformer-based model General Text Embeddings-Large is employed to obtain the embeddings of the utterances from each speaker. Results show that lower similarity is associated with greater positive emotional experiences in the positive (but not conflict) conversation.
  arXiv  Detail & Related papers  (2023-09-22T06:37:45Z)
• Agentivit\`a e telicit\`a in GilBERTo: implicazioni cognitive [77.71680953280436]
  The goal of this study is to investigate whether a Transformer-based neural language model infers lexical semantics. The semantic properties considered are telicity (also combined with definiteness) and agentivity.
  arXiv  Detail & Related papers  (2023-07-06T10:52:22Z)
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey [71.43956423427397]
  We aim to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
  arXiv  Detail & Related papers  (2022-07-20T13:37:57Z)
• Perception Point: Identifying Critical Learning Periods in Speech for Bilingual Networks [58.24134321728942]
  We compare and identify cognitive aspects on deep neural-based visual lip-reading models. We observe a strong correlation between these theories in cognitive psychology and our unique modeling.
  arXiv  Detail & Related papers  (2021-10-13T05:30:50Z)
• A Neural Network-Based Linguistic Similarity Measure for Entrainment in Conversations [12.052672647509732]
  Linguistic entrainment is a phenomenon where people tend to mimic each other in conversation. Most of the current similarity measures are based on bag-of-words approaches. We propose to use a neural network model to perform the similarity measure for entrainment.
  arXiv  Detail & Related papers  (2021-09-04T19:48:17Z)
• Decoupling entrainment from consistency using deep neural networks [14.823143667165382]
  Isolating the effect of consistency, i.e., speakers adhering to their individual styles, is a critical part of the analysis of entrainment. We propose to treat speakers' initial vocal features as confounds for the prediction of subsequent outputs. Using two existing neural approaches to deconfounding, we define new measures of entrainment that control for consistency.
  arXiv  Detail & Related papers  (2020-11-03T17:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.