Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue

• URL: http://arxiv.org/abs/2609.16396v1
• Date: Mon, 14 Sep 2026 22:07:34 GMT
• Title: Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue
• Abstract summary: We study 27 human-human interviews conducted in virtual reality.<n>We compare 20 time-series models while excluding lexical and acoustic information, and then systematically vary temporal context, interactional source, modality availability, and event timing.<n>Across grouped 10-fold cross-validation, the strongest probes reach up to.75 mean held-out AUROC from speaker-side behavior.
• Score: 44.39196719594625
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Negation is typically modeled through its linguistic realization, although spoken interaction is accompanied by tightly coordinated nonverbal behavior. We ask whether contexts centered on spoken negation cues contain measurable multimodal behavioral information: whether they can be distinguished from matched control contexts without lexical or acoustic input, where this information occurs in time, which modalities carry it, and whether it extends to the dialogue partner. We study 27 human-human interviews conducted in virtual reality, comprising temporally aligned gaze, facial, head, body, hand, and finger behavior and 964 annotated negation cues. Treating classification as a predictive probe, we compare 20 time-series models while excluding lexical and acoustic information, and then systematically vary temporal context, interactional source, modality availability, and event timing. Across grouped 10-fold cross-validation, the strongest probes reach up to .75 mean held-out AUROC from speaker-side behavior. Temporal analyses show that predictive information is concentrated around cue onset but remains detectable over a broader surrounding interval, while dialogue-partner behavior carries weaker predictive information with a comparatively diffuse temporal profile. Ablation and timing perturbations further show that facial features produce the largest modality-ablation effect and that the trained probe is sensitive to the temporal organization of the observed events.

Related papers

• TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue [79.83228182492354]
  Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor.<n>We present TurnBench, a benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection.<n>We find end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in backchannel-dense interaction styles.
  arXiv  Detail & Related papers  (2026-08-25T23:14:36Z)
• Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering [87.43777061308658]
  We analyze predictive behavior in full spoken language models (FDSLMs)<n>We show that FDSLMs dynamically modulate their internal focus between two states: a generative state aligned with model output generation and a perceptive state aligned incoming user input.<n>During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it miss the beginning of incoming input.
  arXiv  Detail & Related papers  (2026-06-09T19:08:07Z)
• A Multimodal Framework for Dementia Detection via Linguistic and Acoustic Representation Learning [10.559333552210434]
  Alzheimer's disease is the leading cause of dementia, affecting memory, reasoning, communication, and daily functioning.<n>Recent studies have demonstrated that spontaneous speech contains valuable linguistic and acoustic biomarkers associated with dementia.<n>We propose a multimodal deep learning framework for automatic dementia detection that jointly exploits speech and transcript information in an end-to-end trainable manner.
  arXiv  Detail & Related papers  (2026-05-25T07:57:49Z)
• SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models [86.19617358080016]
  Social Omni is a benchmark that operationalizes the evaluation of conversational interactivity across three core dimensions.<n>Social Omni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances.<n>Our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions.
  arXiv  Detail & Related papers  (2026-03-17T17:58:44Z)
• Coherence in the brain unfolds across separable temporal regimes [1.3874648807526748]
  Coherence in language requires the brain to satisfy two competing temporal demands.<n>We show that coherence is implemented through dissociable neural regimes of slow contextual integration and rapid event-driven reconfiguration.
  arXiv  Detail & Related papers  (2025-12-23T16:16:42Z)
• PersonaDrift: A Benchmark for Temporal Anomaly Detection in Language-Based Dementia Monitoring [0.9668407688201359]
  PersonaDrift is a benchmark designed to evaluate machine learning and statistical methods for detecting progressive changes in daily communication.<n>The benchmark focuses on two forms of longitudinal change that caregivers highlighted as particularly salient: flattened sentiment and off-topic replies.<n>Preliminary results show that flattened sentiment can often be detected with simple statistical models in users with low baseline variability.
  arXiv  Detail & Related papers  (2025-11-20T15:15:00Z)
• Nonverbal Interaction Detection [83.40522919429337]
  This work addresses a new challenge of understanding human nonverbal interaction in social contexts. We contribute a novel large-scale dataset, called NVI, which is meticulously annotated to include bounding boxes for humans and corresponding social groups. Second, we establish a new task NVI-DET for nonverbal interaction detection, which is formalized as identifying triplets in the form individual, group, interaction> from images. Third, we propose a nonverbal interaction detection hypergraph (NVI-DEHR), a new approach that explicitly models high-order nonverbal interactions using hypergraphs.
  arXiv  Detail & Related papers  (2024-07-11T02:14:06Z)
• Dynamic Time-Alignment of Dimensional Annotations of Emotion using Recurrent Neural Networks [9.02236667251654]
  We propose a method that can compensate inconsistencies across annotations and synchronise the traces with the corresponding acoustic features. Results show that our method can significantly increase inter-annotator agreement, as well as correlation between traces and audio features.
  arXiv  Detail & Related papers  (2022-09-21T09:38:57Z)
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey [71.43956423427397]
  We aim to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
  arXiv  Detail & Related papers  (2022-07-20T13:37:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.