Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues

• URL: http://arxiv.org/abs/2603.20079v1
• Date: Fri, 20 Mar 2026 15:58:54 GMT
• Title: Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues
• Abstract summary: We investigate how verbal and nonverbal linguistic features, exhibited by speakers and listeners in dialogue, can contribute to predicting the listener's state of understanding.<n>Specifically, we examine three linguistic cues related to cognitive load and hypothesised to correlate with listener understanding.
• Score: 4.351015250928505
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We investigate how verbal and nonverbal linguistic features, exhibited by speakers and listeners in dialogue, can contribute to predicting the listener's state of understanding in explanatory interactions on a moment-by-moment basis. Specifically, we examine three linguistic cues related to cognitive load and hypothesised to correlate with listener understanding: the information value (operationalised with surprisal) and syntactic complexity of the speaker's utterances, and the variation in the listener's interactive gaze behaviour. Based on statistical analyses of the MUNDEX corpus of face-to-face dialogic board game explanations, we find that individual cues vary with the listener's level of understanding. Listener states ('Understanding', 'Partial Understanding', 'Non-Understanding' and 'Misunderstanding') were self-annotated by the listeners using a retrospective video-recall method. The results of a subsequent classification experiment, involving two off-the-shelf classifiers and a fine-tuned German BERT-based multimodal classifier, demonstrate that prediction of these four states of understanding is generally possible and improves when the three linguistic cues are considered alongside textual features.

Related papers

• Information Dynamics of Language Communication [3.2562256911091407]
  We introduce an information-theoretic framework that quantifies the directed flow of semantic content between interlocutors.<n>Across four experiments we show that the framework detects reduced information flow in cognitively rigid dialogue.<n>This framework opens new avenues for studying information dynamics in digital discourse, pedagogical interactions, clinical dialogues, and any domain in which the structure of linguistic exchange is of research relevance.
  arXiv  Detail & Related papers  (2026-06-29T10:31:58Z)
• From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interaction [25.529747060487363]
  SpeechLLMs process spoken language directly from audio, but accent and vocal identity cues can lead to biased behaviour.<n>We distinguish quality-of-service disparities from content-level bias in coherent outputs and examine intersectional effects of accent and perceived gender.
  arXiv  Detail & Related papers  (2026-03-19T21:12:49Z)
• Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue [12.181747090385612]
  Mental manipulation is the strategic use of language to covertly influence or exploit others.<n>We present the first study of mental manipulation detection in spoken dialogues.<n>Using few-shot large audio-language models and human annotation, we evaluate how modality affects detection accuracy and perception.
  arXiv  Detail & Related papers  (2026-01-13T09:02:08Z)
• Computational Analysis of Conversation Dynamics through Participant Responsivity [18.116125865284666]
  We develop and evaluate methods for quantifying responsivity.<n>We evaluate both methods against a ground truth set of human-annotated conversations.<n>We then develop conversation-level derived metrics to address various aspects of conversational discourse.
  arXiv  Detail & Related papers  (2025-09-19T23:13:13Z)
• SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding Large Language Models [76.07833875692722]
  Speech-based Intelligence Quotient (SIQ) is a new form of human cognition-inspired evaluation pipeline for voice understanding large language models.<n>Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks.
  arXiv  Detail & Related papers  (2025-07-25T15:12:06Z)
• MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions [70.93364531054273]
  We introduce MultiVox, the first benchmark to evaluate the ability of voice assistants to integrate spoken and visual cues.<n>Specifically, MultiVox includes 1000 human-annotated and recorded speech dialogues that encompass diverse paralinguistic features.<n>Our evaluation on 10 state-of-the-art models reveals that, although humans excel at these tasks, current models consistently struggle to produce contextually grounded responses.
  arXiv  Detail & Related papers  (2025-07-14T23:20:42Z)
• Speaker effects in spoken language comprehension [0.9514940899499753]
  The identity of a speaker significantly influences spoken language comprehension by affecting both perception and expectation.<n>We propose an integrative model featuring the interplay between bottom-up perception-based processes driven by acoustic details and top-down expectation-based processes driven by a speaker model.
  arXiv  Detail & Related papers  (2024-12-10T07:03:06Z)
• Improving Speaker Diarization using Semantic Information: Joint Pairwise Constraints Propagation [53.01238689626378]
  We propose a novel approach to leverage semantic information in speaker diarization systems. We introduce spoken language understanding modules to extract speaker-related semantic information. We present a novel framework to integrate these constraints into the speaker diarization pipeline.
  arXiv  Detail & Related papers  (2023-09-19T09:13:30Z)
• Can Language Models Learn to Listen? [96.01685069483025]
  We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words. Our approach autoregressively predicts a response of a listener: a sequence of listener facial gestures, quantized using a VQ-VAE. We show that our generated listener motion is fluent and reflective of language semantics through quantitative metrics and a qualitative user study.
  arXiv  Detail & Related papers  (2023-08-21T17:59:02Z)
• Learning to Mediate Disparities Towards Pragmatic Communication [9.321336642983875]
  We propose Pragmatic Rational Speaker (PRS) as a framework for building AI agents with similar abilities in language communication. The PRS attempts to learn the speaker-listener disparity and adjust the speech accordingly, by adding a light-weighted disparity adjustment layer into working memory. By fixing the long-term memory, the PRS only needs to update its working memory to learn and adapt to different types of listeners.
  arXiv  Detail & Related papers  (2022-03-25T14:46:43Z)
• Perception Point: Identifying Critical Learning Periods in Speech for Bilingual Networks [58.24134321728942]
  We compare and identify cognitive aspects on deep neural-based visual lip-reading models. We observe a strong correlation between these theories in cognitive psychology and our unique modeling.
  arXiv  Detail & Related papers  (2021-10-13T05:30:50Z)
• What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Study [6.696983725360809]
  Language Models (LMs) have been ubiquitously leveraged in various tasks including spoken language understanding (SLU) In this work, we propose to dissect SLU into three representative properties: speakeral (disfluency, pause, overtalk), channel (conversation-type, turn-tasks) and ASR (insertion, deletion,substitution) We probe BERT based language models (BERT, RoBERTa) trained on spoken transcripts to investigate its ability to understand multifarious properties in absence of any speech cues.
  arXiv  Detail & Related papers  (2021-09-19T11:23:50Z)
• Improving Machine Reading Comprehension with Contextualized Commonsense Knowledge [62.46091695615262]
  We aim to extract commonsense knowledge to improve machine reading comprehension. We propose to represent relations implicitly by situating structured knowledge in a context. We employ a teacher-student paradigm to inject multiple types of contextualized knowledge into a student machine reader.
  arXiv  Detail & Related papers  (2020-09-12T17:20:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.