Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings

• URL: http://arxiv.org/abs/2606.17542v1
• Date: Tue, 16 Jun 2026 05:46:44 GMT
• Title: Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings
• Abstract summary: We investigate turn-taking in multimodal multi-party conversations using large language models (LLMs)<n>We construct an evaluation framework for three tasks: addressee detection, turn-change prediction, and next speaker prediction.<n>We compare supervised models trained for these tasks, text-based LLMs, multimodal LLMs (MM-LLMs), and human subjects.
• Score: 85.35443905857876
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate turn-taking in multimodal multi-party conversations using large language models (LLMs). We construct an evaluation framework for three tasks: addressee detection, turn-change prediction, and next speaker prediction. We compare supervised models trained for these tasks, text-based LLMs, multimodal LLMs (MM-LLMs), and human subjects. Experiments on the AMI corpus showed that LLMs outperformed supervised models and humans in next speaker prediction, despite not being trained on the target domain and without access to audio or visual information. An MM-LLM performed better than text-based LLMs on addressee detection and turn-change prediction but remained below human performance, indicating difficulty leveraging raw audio-visual signals. Ablation analyses revealed that conversational context was critical, particularly for next speaker prediction. We observed that human and LLM prediction patterns were similar, and intervals with frequent turn changes were difficult for both.

Related papers

• SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Prediction? [42.890703748349814]
  We show that even leading models surprisingly struggle to predict a human speaker's next utterance.<n>Instead, humans can readily anticipate forthcoming utterances based on multimodal cues, such as gestures, gaze, and emotional tone, from the context.<n>We develop a dual-route prediction MLLM, SayNext-Chat, that incorporates cognitively inspired design to emulate predictive processing in conversation.
  arXiv  Detail & Related papers  (2026-01-30T21:27:45Z)
• ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction [88.41471266579333]
  We propose ELEGANCE, a novel framework that incorporates linguistic knowledge from large language models (LLMs) into AV-TSE models.<n> Comprehensive experiments with RoBERTa, Qwen3-0.6B, and Qwen3-4B on two AV-TSE backbones show significant improvements.
  arXiv  Detail & Related papers  (2025-11-09T08:50:11Z)
• Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction [87.49303116989708]
  We explore the potential of pre-trained speech-language models (PSLMs) and pre-trained language models (PLMs) as auxiliary knowledge sources for AV-TSE.<n>In this study, we propose incorporating the linguistic constraints from PSLMs or PLMs for the AV-TSE model as additional supervision signals.<n>Without any extra computational cost during inference, the proposed approach consistently improves speech quality and intelligibility.
  arXiv  Detail & Related papers  (2025-06-11T14:36:26Z)
• Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning [30.308743810639758]
  Large audio language models (LALMs) have to be evaluated on reasoning related tasks which are different from traditional classification or generation tasks.<n>We benchmark open-source LALMs and observe that they are consistently behind human capabilities on the tasks in the TREA dataset.<n>Our analysis shows that the accuracy and uncertainty metrics are not necessarily correlated and thus, points to a need for wholesome evaluation of LALMs for high-stakes applications.
  arXiv  Detail & Related papers  (2025-05-19T13:46:35Z)
• Lla-VAP: LSTM Ensemble of Llama and VAP for Turn-Taking Prediction [0.0]
  This project expands on existing strategies for turn-taking prediction by employing a multi-modal ensemble approach.<n>We aim to improve the accuracy and efficiency of identifying TRPs in both scripted and unscripted conversational scenarios.
  arXiv  Detail & Related papers  (2024-12-24T00:20:38Z)
• LLMs are Superior Feedback Providers: Bootstrapping Reasoning for Lie Detection with Self-Generated Feedback [33.14770105185958]
  Large Language Models (LLMs) excel at generating human-like dialogues and comprehending text. We propose a bootstrapping framework that leverages self-generated feedback to enhance LLM reasoning capabilities for lie detection. We investigate the application of the proposed framework for detecting betrayal and deception in Diplomacy games, and compare it with feedback from professional human players.
  arXiv  Detail & Related papers  (2024-08-25T18:47:55Z)
• Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion [38.78341787348164]
  We propose an approach for continuous prediction of turn-taking and backchanneling locations in spoken dialogue by fusing a neural acoustic model with a large language model (LLM) Experiments on the Switchboard human-human conversation dataset demonstrate that our approach consistently outperforms the baseline models with single modality.
  arXiv  Detail & Related papers  (2024-01-26T08:59:07Z)
• Generative Context-aware Fine-tuning of Self-supervised Speech Models [54.389711404209415]
  We study the use of generative large language models (LLM) generated context information. We propose an approach to distill the generated information during fine-tuning of self-supervised speech models. We evaluate the proposed approach using the SLUE and Libri-light benchmarks for several downstream tasks: automatic speech recognition, named entity recognition, and sentiment analysis.
  arXiv  Detail & Related papers  (2023-12-15T15:46:02Z)
• Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational Models [0.8139163264824348]
  Humans are known to use salient multimodal features, such as visual cues, to facilitate the processing of upcoming words. multimodal computational models can integrate visual and linguistic data using a visual attention mechanism to assign next-word probabilities. We show that predictability estimates from humans aligned more closely with scores generated from multimodal models vs. their unimodal counterparts.
  arXiv  Detail & Related papers  (2023-08-11T09:30:07Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech [67.88748572167309]
  We present LDNet, a unified framework for mean opinion score (MOS) prediction. We propose two inference methods that provide more stable results and efficient computation.
  arXiv  Detail & Related papers  (2021-10-18T08:52:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.