Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement

• URL: http://arxiv.org/abs/2412.05797v1
• Date: Sun, 08 Dec 2024 03:26:44 GMT
• Title: Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement
• Abstract summary: multimodal analytics is crucial for identifying non-verbal interactions of group members.<n>In this demo, we illustrate our present capabilities at detecting and tracking nonverbal behavior in student task-oriented interactions in the classroom.
• Score: 17.25829281965904
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Our goal is to develop an AI Partner that can provide support for group problem solving and social dynamics. In multi-party working group environments, multimodal analytics is crucial for identifying non-verbal interactions of group members. In conjunction with their verbal participation, this creates an holistic understanding of collaboration and engagement that provides necessary context for the AI Partner. In this demo, we illustrate our present capabilities at detecting and tracking nonverbal behavior in student task-oriented interactions in the classroom, and the implications for tracking common ground and engagement.

Related papers

• Informal Learning Emerges in Everyday Human-LLM Interaction [74.53081471113929]
  We show that everyday human-LLM interaction contains measurable, selective and conditionally organized behavioural signatures of informal learning.<n>They shift AI evaluation from answer-delivery efficiency toward the preservation of cognitive opportunities for users to reason, test ideas and construct understanding.
  arXiv  Detail & Related papers  (2026-07-20T07:56:41Z)
• CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views [82.15362247351763]
  We introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios.<n>The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects.<n>We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction.
  arXiv  Detail & Related papers  (2026-07-07T18:08:45Z)
• Bridging Talk and Thought: Understanding Dialogue Dynamics Across Collaborative Problem-Solving Contexts [57.79777631004424]
  We present a conceptual framework for analyzing dialogue in collaborative problem-solving contexts.<n>Our framework integrates cognitive and non-cognitive problem solving with metacognitive regulatory mechanisms.
  arXiv  Detail & Related papers  (2026-06-25T16:20:49Z)
• Learning When to Cooperate Under Heterogeneous Goals [4.265773997354609]
  We extend the typical Ad Hoc Teamwork (AHT) setting to incorporate the idea of agents having heterogeneous goals that in any given scenario may or may not overlap.<n>We introduce a novel approach to learning policies in this setting, based on a hierarchical combination of imitation and reinforcement learning.<n>We also investigate the contribution of an auxiliary component that learns to model teammates by predicting their actions, finding that its effect on performance is inversely related to the amount of observable information about teammate goals.
  arXiv  Detail & Related papers  (2026-03-07T15:09:27Z)
• Conversational Learning Diagnosis via Reasoning Multi-Turn Interactive Learning [16.641654527654246]
  ParLD is a preview-analyze-reason framework for conversational learning diagnosis.<n>It uses multi-agent collaboration to diagnose students' cognitive state over multiple dialogue turns.
  arXiv  Detail & Related papers  (2026-03-03T18:29:00Z)
• The Rise of AI Agent Communities: Large-Scale Analysis of Discourse and Interaction on Moltbook [62.2627874717318]
  Moltbook is a Reddit-like social platform where AI agents create posts and interact with other agents through comments and replies.<n>Using a public API snapshot collected about five days after launch, we address three research questions: what AI agents discuss, how they post, and how they interact.<n>We show that agents' writing is predominantly neutral, with positivity appearing in community engagement and assistance-oriented content.
  arXiv  Detail & Related papers  (2026-02-13T05:28:31Z)
• Measuring Inclusion in Interaction: Inclusion Analytics for Human-AI Collaborative Learning [1.0742675209112622]
  We introduce inclusion analytics, a discourse-based framework for examining inclusion as a dynamic, interactional process in problem solving.<n>We demonstrate how these constructs can be made analytically visible using scalable, interaction-level measures.<n>This work represents an initial step toward process-oriented approaches to measuring inclusion in human-AI collaborative learning environments.
  arXiv  Detail & Related papers  (2026-02-09T23:07:15Z)
• The Impact of Background Speech on Interruption Detection in Collaborative Groups [4.413246337852144]
  We analyze interruption detection in single-conversation and multi-group dialogue settings.<n>We then create a state-of-the-art method for interruption identification that is robust to overlapping speech.<n>Our work highlights meaningful linguistic and prosodic information about how interruptions manifest in collaborative group interactions.
  arXiv  Detail & Related papers  (2025-07-09T20:57:55Z)
• Personalizing Student-Agent Interactions Using Log-Contextualized Retrieval Augmented Generation (RAG) [3.5262811753348715]
  Large language models (LLMs) facilitate dynamic pedagogical interactions but undermine confidence, trust, and instructional value.<n>Retrieval-augmented generation (RAG) grounds LLM outputs in curated knowledge but requires a clear semantic link between user input and a knowledge base.<n>We propose log-contextualized RAG (LC-RAG), which enhances RAG retrieval by using environment logs to contextualize collaborative discourse.
  arXiv  Detail & Related papers  (2025-05-22T19:31:40Z)
• Nonverbal Interaction Detection [83.40522919429337]
  This work addresses a new challenge of understanding human nonverbal interaction in social contexts. We contribute a novel large-scale dataset, called NVI, which is meticulously annotated to include bounding boxes for humans and corresponding social groups. Second, we establish a new task NVI-DET for nonverbal interaction detection, which is formalized as identifying triplets in the form individual, group, interaction> from images. Third, we propose a nonverbal interaction detection hypergraph (NVI-DEHR), a new approach that explicitly models high-order nonverbal interactions using hypergraphs.
  arXiv  Detail & Related papers  (2024-07-11T02:14:06Z)
• Harnessing Transparent Learning Analytics for Individualized Support through Auto-detection of Engagement in Face-to-Face Collaborative Learning [3.0184625301151833]
  This paper proposes a transparent approach to automatically detect student's individual engagement in the process of collaboration. The proposed approach can reflect student's individual engagement and can be used as an indicator to distinguish students with different collaborative learning challenges.
  arXiv  Detail & Related papers  (2024-01-03T12:20:28Z)
• Understanding Idea Creation in Collaborative Discourse through Networks: The Joint Attention-Interaction-Creation (AIC) Framework [0.42303492200814446]
  The Joint Attention-Interaction-Creation (AIC) framework captures important dynamics in collaborative discourse. The framework was developed from the networked lens, informed by natural language processing techniques, and inspired by socio-semantic network analysis.
  arXiv  Detail & Related papers  (2023-05-25T17:18:19Z)
• From Interactive to Co-Constructive Task Learning [13.493719155524404]
  We will review current proposals for interactive task learning and discuss their main contributions. We then discuss our notion of co-construction and summarize research insights from adult-child and human-robot interactions.
  arXiv  Detail & Related papers  (2023-05-24T19:45:30Z)
• Interactive Natural Language Processing [67.87925315773924]
  Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP. This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
  arXiv  Detail & Related papers  (2023-05-22T17:18:29Z)
• A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects [100.75759050696355]
  We provide a comprehensive overview of the prominent problems and advanced designs for conversational agent's proactivity in different types of dialogues. We discuss challenges that meet the real-world application needs but require a greater research focus in the future.
  arXiv  Detail & Related papers  (2023-05-04T11:38:49Z)
• Automatic Context-Driven Inference of Engagement in HMI: A Survey [6.479224589451863]
  This paper presents a survey on engagement inference for human-machine interaction. It entails interdisciplinary definition, engagement components and factors, publicly available datasets, ground truth assessment, and most commonly used features and methods. It serves as a guide for the development of future human-machine interaction interfaces with reliable context-aware engagement inference capability.
  arXiv  Detail & Related papers  (2022-09-30T10:46:13Z)
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey [71.43956423427397]
  We aim to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
  arXiv  Detail & Related papers  (2022-07-20T13:37:57Z)
• Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy [83.61404191470126]
  We propose a new solution named I-Pro that can learn Proactive policy in the Interactive setting. Specifically, we learn the trade-off via a learned goal weight, which consists of four factors. The experimental results demonstrate I-Pro significantly outperforms baselines in terms of effectiveness and interpretability.
  arXiv  Detail & Related papers  (2022-04-07T14:11:31Z)
• Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations [59.608216900601384]
  We study agents that learn to communicate via actuating their joints in a 3D environment. We show that under realistic assumptions, a non-uniform distribution of intents and a common-knowledge energy cost, these agents can find protocols that generalize to novel partners.
  arXiv  Detail & Related papers  (2020-10-29T19:23:10Z)
• Towards Effective Human-AI Collaboration in GUI-Based Interactive Task Learning Agents [29.413358312233253]
  We argue that a key challenge in enabling usable and useful interactive task learning for intelligent agents is to facilitate effective Human-AI collaboration. We reflect on our past 5 years of efforts on designing, developing and studying the SUGILITE system.
  arXiv  Detail & Related papers  (2020-03-05T14:12:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.