Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement

• URL: http://arxiv.org/abs/2412.05797v1
• Date: Sun, 08 Dec 2024 03:26:44 GMT
• Title: Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement
• Authors: Derek Palmer, Yifan Zhu, Kenneth Lai, Hannah VanderHoeven, Mariah Bradford, Ibrahim Khebour, Carlos Mabrey, Jack Fitzgerald, Nikhil Krishnaswamy, Martha Palmer, James Pustejovsky,
• Abstract summary: multimodal analytics is crucial for identifying non-verbal interactions of group members. In this demo, we illustrate our present capabilities at detecting and tracking nonverbal behavior in student task-oriented interactions in the classroom.
• Score: 17.25829281965904
• License:
• Abstract: Our goal is to develop an AI Partner that can provide support for group problem solving and social dynamics. In multi-party working group environments, multimodal analytics is crucial for identifying non-verbal interactions of group members. In conjunction with their verbal participation, this creates an holistic understanding of collaboration and engagement that provides necessary context for the AI Partner. In this demo, we illustrate our present capabilities at detecting and tracking nonverbal behavior in student task-oriented interactions in the classroom, and the implications for tracking common ground and engagement.

Related papers

• Nonverbal Interaction Detection [83.40522919429337]
  This work addresses a new challenge of understanding human nonverbal interaction in social contexts. We contribute a novel large-scale dataset, called NVI, which is meticulously annotated to include bounding boxes for humans and corresponding social groups. Second, we establish a new task NVI-DET for nonverbal interaction detection, which is formalized as identifying triplets in the form individual, group, interaction> from images. Third, we propose a nonverbal interaction detection hypergraph (NVI-DEHR), a new approach that explicitly models high-order nonverbal interactions using hypergraphs.
  arXiv  Detail & Related papers  (2024-07-11T02:14:06Z)
• Harnessing Transparent Learning Analytics for Individualized Support through Auto-detection of Engagement in Face-to-Face Collaborative Learning [3.0184625301151833]
  This paper proposes a transparent approach to automatically detect student's individual engagement in the process of collaboration. The proposed approach can reflect student's individual engagement and can be used as an indicator to distinguish students with different collaborative learning challenges.
  arXiv  Detail & Related papers  (2024-01-03T12:20:28Z)
• Understanding Idea Creation in Collaborative Discourse through Networks: The Joint Attention-Interaction-Creation (AIC) Framework [0.42303492200814446]
  The Joint Attention-Interaction-Creation (AIC) framework captures important dynamics in collaborative discourse. The framework was developed from the networked lens, informed by natural language processing techniques, and inspired by socio-semantic network analysis.
  arXiv  Detail & Related papers  (2023-05-25T17:18:19Z)
• From Interactive to Co-Constructive Task Learning [13.493719155524404]
  We will review current proposals for interactive task learning and discuss their main contributions. We then discuss our notion of co-construction and summarize research insights from adult-child and human-robot interactions.
  arXiv  Detail & Related papers  (2023-05-24T19:45:30Z)
• Interactive Natural Language Processing [67.87925315773924]
  Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP. This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
  arXiv  Detail & Related papers  (2023-05-22T17:18:29Z)
• A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects [100.75759050696355]
  We provide a comprehensive overview of the prominent problems and advanced designs for conversational agent's proactivity in different types of dialogues. We discuss challenges that meet the real-world application needs but require a greater research focus in the future.
  arXiv  Detail & Related papers  (2023-05-04T11:38:49Z)
• Automatic Context-Driven Inference of Engagement in HMI: A Survey [6.479224589451863]
  This paper presents a survey on engagement inference for human-machine interaction. It entails interdisciplinary definition, engagement components and factors, publicly available datasets, ground truth assessment, and most commonly used features and methods. It serves as a guide for the development of future human-machine interaction interfaces with reliable context-aware engagement inference capability.
  arXiv  Detail & Related papers  (2022-09-30T10:46:13Z)
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey [71.43956423427397]
  We aim to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
  arXiv  Detail & Related papers  (2022-07-20T13:37:57Z)
• Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy [83.61404191470126]
  We propose a new solution named I-Pro that can learn Proactive policy in the Interactive setting. Specifically, we learn the trade-off via a learned goal weight, which consists of four factors. The experimental results demonstrate I-Pro significantly outperforms baselines in terms of effectiveness and interpretability.
  arXiv  Detail & Related papers  (2022-04-07T14:11:31Z)
• Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations [59.608216900601384]
  We study agents that learn to communicate via actuating their joints in a 3D environment. We show that under realistic assumptions, a non-uniform distribution of intents and a common-knowledge energy cost, these agents can find protocols that generalize to novel partners.
  arXiv  Detail & Related papers  (2020-10-29T19:23:10Z)
• Towards Effective Human-AI Collaboration in GUI-Based Interactive Task Learning Agents [29.413358312233253]
  We argue that a key challenge in enabling usable and useful interactive task learning for intelligent agents is to facilitate effective Human-AI collaboration. We reflect on our past 5 years of efforts on designing, developing and studying the SUGILITE system.
  arXiv  Detail & Related papers  (2020-03-05T14:12:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.